United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
JDK-8027029 : Deadlock in caching code launching application with a large number of jars (~100).

Details
Type:
Bug
Submit Date:
2013-10-22
Status:
Closed
Updated Date:
2014-01-13
Project Name:
JDK
Resolved Date:
2013-11-04
Component:
deploy
OS:
windows_7
Sub-Component:
plugin
CPU:
Priority:
P2
Resolution:
Fixed
Affected Versions:
7u45
Fixed Versions:
7u51 (b06)

Related Reports
Backport:
Backport:
Backport:
Duplicate:

Sub Tasks

Description
FULL PRODUCT VERSION :
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) Client VM (build 24.45-b08, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows [Version 6.1.7601]


A DESCRIPTION OF THE PROBLEM :
deadlock launching jnlp application.

REGRESSION.  Last worked in version 7u40

ADDITIONAL REGRESSION INFORMATION:
1.7.0_40.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
create a large jnlp application.  i.e. a netbeans platform app launched via webstart.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
launches.
ACTUAL -
deadlocks.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
Found one Java-level deadlock:
=============================
"Java Web Start Main Thread":
  waiting to lock monitor 0x066368bc (object 0x29e65428, a java.lang.Class),
  which is held by "Finalizer"
"Finalizer":
  waiting to lock monitor 0x00dbc954 (object 0x2a1b52b0, a com.sun.deploy.cache.CacheEntry),
  which is held by "Java Web Start Main Thread"

Java stack information for the threads listed above:
===================================================
"Java Web Start Main Thread":
    at com.sun.deploy.cache.MemoryCache.getLoadedResource(Unknown Source)
    - waiting to lock <0x29e65428> (a java.lang.Class for com.sun.deploy.cache.MemoryCache)
    at com.sun.deploy.cache.CachedJarFile.getCacheEntry(Unknown Source)
    - locked <0x24810058> (a com.sun.deploy.cache.CachedJarFile)
    at com.sun.deploy.cache.CachedJarFile.getManifest(Unknown Source)
    - locked <0x24810058> (a com.sun.deploy.cache.CachedJarFile)
    at com.sun.deploy.util.JarUtil.isBlobSigned(Unknown Source)
    at com.sun.deploy.cache.CacheEntry.getJarFile(Unknown Source)
    - locked <0x2a1b52b0> (a com.sun.deploy.cache.CacheEntry)
    at com.sun.deploy.security.DeployManifestChecker.verify(Unknown Source)
    at com.sun.javaws.security.AppPolicy.grantUnrestrictedAccess(Unknown Source)
    at com.sun.javaws.security.JNLPSignedResourcesHelper.checkSignedResourcesHelper(Unknown Source)
    at com.sun.javaws.security.JNLPSignedResourcesHelper.checkSignedResources(Unknown Source)
    at com.sun.javaws.Launcher.prepareResources(Unknown Source)
    at com.sun.javaws.Launcher.prepareAllResources(Unknown Source)
    at com.sun.javaws.Launcher.prepareToLaunch(Unknown Source)
    at com.sun.javaws.Launcher.prepareToLaunch(Unknown Source)
    at com.sun.javaws.Launcher.launch(Unknown Source)
    at com.sun.javaws.Main.launchApp(Unknown Source)
    at com.sun.javaws.Main.continueInSecureThread(Unknown Source)
    at com.sun.javaws.Main.access$000(Unknown Source)
    at com.sun.javaws.Main$1.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
"Finalizer":
    at com.sun.deploy.cache.CacheEntry.storageFilesExist(Unknown Source)
    - waiting to lock <0x2a1b52b0> (a com.sun.deploy.cache.CacheEntry)
    at com.sun.deploy.cache.MemoryCache.validateResource(Unknown Source)
    at com.sun.deploy.cache.MemoryCache.getLoadedResource(Unknown Source)
    - locked <0x29e65428> (a java.lang.Class for com.sun.deploy.cache.MemoryCache)
    at com.sun.deploy.cache.CachedJarFile.isReferencedTo(Unknown Source)
    at com.sun.deploy.cache.CachedJarFile.close(Unknown Source)
    at java.util.zip.ZipFile.finalize(Unknown Source)
    at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
    at java.lang.ref.Finalizer.runFinalizer(Unknown Source)
    at java.lang.ref.Finalizer.access$100(Unknown Source)
    at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

Found 1 deadlock.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
the deadlock seems to be from new code added to getJarFile() in com.sun.deploy.cache.CacheEntry in deploy.jar between 7u40 and 7u45.

are you TRYING to kill webstart?  i'm sure you're not, but it sure seems that way since 6u19. </rant!>



---------- END SOURCE ----------
                                    

Comments
SQE OK to take the regression fix into CPU14_01
                                     
2013-11-04
7u51 justification

	why this is critical for 7u51
		this is a serious regression introduced in latest 7u45 release, application hangs at startup
		
	what is the risk in taking the fix
		the risk of the fix is low, the fix removes unnecessary synchronization by moving existing security check from sync block

	what are the implications if we don???t take it
		this deadlock is reproducible with large applications - see thread on stackoverflow from comments above
		if we don't fix it in 7u51, people will continue to face this issue (without any known workaround)

                                     
2013-10-31
review request in crucible for backport 7u51 changes - https://sthinfra10.se.oracle.com/cru/CR-JDK7UCPU-160
                                     
2013-10-31
review thread - http://sa.sfbay.sun.com/mail-archive/8027029-deployment/
                                     
2013-10-30
We had similar issues before - see 7191454 - we should avoid holding CacheEntry lock while accessing MemoryCache synchronized methods.
                                     
2013-10-25
This hang caused by the fix for 8025723 and by the following changes in CacheEntry

    public JarFile getJarFile() {
        ...
        synchronized(this) {
            ...
+                    if (hasSigningInfo() && !JarUtil.isBlobSigned(newJarFile)
+                               && !JarUtil.allJarEntriesSigned(newJarFile)) {
+                          throw new SecurityException(...)
            ...
        }

The launcher thread holds CacheEntry lock and attempts to acquire MemoryCache.class lock.
                                     
2013-10-25
This is clearly a java level deadlock in CacheEntry vs MemoryCache.
CacheEntry uses mixture of both synchronized methods, and synchornized(this) blocks, all locking on the CacheEntry instance object.
MemoryCache also has several synchronized methods, and these are both called by and call into CacheEntry to cause the deadlock.
This is a higher priority than P3, and upping to P2.
                                     
2013-10-23
This may be a dup of this thread on stackoverflow:
http://stackoverflow.com/questions/19505505/new-deadlock-bug-in-7u45-webstart
                                     
2013-10-22



Hardware and Software, Engineered to Work Together