Bug ID: JDK-8027029 Deadlock in caching code launching application with a large number of jars (~100).

Type: Bug
Component: deploy
Sub-Component: plugin
Affected Version: 7u45

Priority: P2
Status: Closed
Resolution: Fixed
OS: windows_7

Submitted: 2013-10-22
Updated: 2014-02-12
Resolved: 2013-11-04

JDK 7	JDK 8
7u51 b06Fixed	8Fixed

FULL PRODUCT VERSION :
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) Client VM (build 24.45-b08, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows [Version 6.1.7601]


A DESCRIPTION OF THE PROBLEM :
deadlock launching jnlp application.

REGRESSION.  Last worked in version 7u40

ADDITIONAL REGRESSION INFORMATION:
1.7.0_40.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
create a large jnlp application.  i.e. a netbeans platform app launched via webstart.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
launches.
ACTUAL -
deadlocks.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
Found one Java-level deadlock:
=============================
"Java Web Start Main Thread":
  waiting to lock monitor 0x066368bc (object 0x29e65428, a java.lang.Class),
  which is held by "Finalizer"
"Finalizer":
  waiting to lock monitor 0x00dbc954 (object 0x2a1b52b0, a com.sun.deploy.cache.CacheEntry),
  which is held by "Java Web Start Main Thread"

Java stack information for the threads listed above:
===================================================
"Java Web Start Main Thread":
    at com.sun.deploy.cache.MemoryCache.getLoadedResource(Unknown Source)
    - waiting to lock <0x29e65428> (a java.lang.Class for com.sun.deploy.cache.MemoryCache)
    at com.sun.deploy.cache.CachedJarFile.getCacheEntry(Unknown Source)
    - locked <0x24810058> (a com.sun.deploy.cache.CachedJarFile)
    at com.sun.deploy.cache.CachedJarFile.getManifest(Unknown Source)
    - locked <0x24810058> (a com.sun.deploy.cache.CachedJarFile)
    at com.sun.deploy.util.JarUtil.isBlobSigned(Unknown Source)
    at com.sun.deploy.cache.CacheEntry.getJarFile(Unknown Source)
    - locked <0x2a1b52b0> (a com.sun.deploy.cache.CacheEntry)
    at com.sun.deploy.security.DeployManifestChecker.verify(Unknown Source)
    at com.sun.javaws.security.AppPolicy.grantUnrestrictedAccess(Unknown Source)
    at com.sun.javaws.security.JNLPSignedResourcesHelper.checkSignedResourcesHelper(Unknown Source)
    at com.sun.javaws.security.JNLPSignedResourcesHelper.checkSignedResources(Unknown Source)
    at com.sun.javaws.Launcher.prepareResources(Unknown Source)
    at com.sun.javaws.Launcher.prepareAllResources(Unknown Source)
    at com.sun.javaws.Launcher.prepareToLaunch(Unknown Source)
    at com.sun.javaws.Launcher.prepareToLaunch(Unknown Source)
    at com.sun.javaws.Launcher.launch(Unknown Source)
    at com.sun.javaws.Main.launchApp(Unknown Source)
    at com.sun.javaws.Main.continueInSecureThread(Unknown Source)
    at com.sun.javaws.Main.access$000(Unknown Source)
    at com.sun.javaws.Main$1.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
"Finalizer":
    at com.sun.deploy.cache.CacheEntry.storageFilesExist(Unknown Source)
    - waiting to lock <0x2a1b52b0> (a com.sun.deploy.cache.CacheEntry)
    at com.sun.deploy.cache.MemoryCache.validateResource(Unknown Source)
    at com.sun.deploy.cache.MemoryCache.getLoadedResource(Unknown Source)
    - locked <0x29e65428> (a java.lang.Class for com.sun.deploy.cache.MemoryCache)
    at com.sun.deploy.cache.CachedJarFile.isReferencedTo(Unknown Source)
    at com.sun.deploy.cache.CachedJarFile.close(Unknown Source)
    at java.util.zip.ZipFile.finalize(Unknown Source)
    at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
    at java.lang.ref.Finalizer.runFinalizer(Unknown Source)
    at java.lang.ref.Finalizer.access$100(Unknown Source)
    at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

Found 1 deadlock.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
the deadlock seems to be from new code added to getJarFile() in com.sun.deploy.cache.CacheEntry in deploy.jar between 7u40 and 7u45.

are you TRYING to kill webstart?  i'm sure you're not, but it sure seems that way since 6u19. </rant!>



---------- END SOURCE ----------

SQE OK to take the regression fix into CPU14_01
04-11-2013
review request in crucible for backport 7u51 changes - https://sthinfra10.se.oracle.com/cru/CR-JDK7UCPU-160
31-10-2013
7u51 justification why this is critical for 7u51 this is a serious regression introduced in latest 7u45 release, application hangs at startup what is the risk in taking the fix the risk of the fix is low, the fix removes unnecessary synchronization by moving existing security check from sync block what are the implications if we don��t take it this deadlock is reproducible with large applications - see thread on stackoverflow from comments above if we don't fix it in 7u51, people will continue to face this issue (without any known workaround)
31-10-2013
review thread - http://sa.sfbay.sun.com/mail-archive/8027029-deployment/
30-10-2013
This hang caused by the fix for 8025723 and by the following changes in CacheEntry public JarFile getJarFile() { ... synchronized(this) { ... + if (hasSigningInfo() && !JarUtil.isBlobSigned(newJarFile) + && !JarUtil.allJarEntriesSigned(newJarFile)) { + throw new SecurityException(...) ... } The launcher thread holds CacheEntry lock and attempts to acquire MemoryCache.class lock.
25-10-2013
We had similar issues before - see 7191454 - we should avoid holding CacheEntry lock while accessing MemoryCache synchronized methods.
25-10-2013
This is clearly a java level deadlock in CacheEntry vs MemoryCache. CacheEntry uses mixture of both synchronized methods, and synchornized(this) blocks, all locking on the CacheEntry instance object. MemoryCache also has several synchronized methods, and these are both called by and call into CacheEntry to cause the deadlock. This is a higher priority than P3, and upping to P2.
23-10-2013
This may be a dup of this thread on stackoverflow: http://stackoverflow.com/questions/19505505/new-deadlock-bug-in-7u45-webstart
22-10-2013