JDK-8096612 : [WebView] "outstanding resource locks detected" when using clip in canvas
  • Type: Bug
  • Component: javafx
  • Sub-Component: web
  • Affected Version: 8u40,8u60
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2015-06-03
  • Updated: 2015-06-12
  • Resolved: 2015-06-11
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 8 JDK 9
8u60Fixed 9Fixed
Related Reports
Relates :  
JDK-8097324 - Error : Outstanding resource locks detected
Relates :  
JDK-8095787 - [WebView] HTML canvas clip() functionality broken
Description
When "clip" is used while painting on an HTML5 canvas, the following msg is printed:

     [java] Outstanding resource locks detected:
     [java] ES2 Vram Pool: 25,735,760 used (9.6%), 25,735,760 managed (9.6%), 268,435,456 total
     [java] 9 total resources being managed
     [java] average resource age is 0.4 frames
     [java] 0 resources at maximum supported age (0.0%)
     [java] 5 resources marked permanent (55.6%)
     [java] 1 resources have had mismatched locks (11.1%)
     [java] 1 resources locked (11.1%)
     [java] 6 resources contain interesting data (66.7%)
     [java] 0 resources disappeared (0.0%)

An html test case is attached.
Comments
Yes, there is a Jira issue somewhere to keep the FX Canvas render ops around as well, but I think in both cases the queue could be the result of an unending animation and unless they periodically erase the entire canvas dimensions (which we detect in FX Canvas), then that could be an unbounded amount of rendering data. Another option is what we do in NGCanvas when the render buffer is tagged "isVolatile" - in that case we save aside the pixels of the buffer after every buffer-full of rendering ops and it costs quite a bit of performance, but it is likely what you'll have to do here as well - either that or remember the ops. I suppose when we add "remember the ops" capability to NGCanvas we can then decide whether to read back the pixels depending on whether it takes less memory to remember the ops vs. the pixels. In the case of these web view buffers, aren't the ones that you are remembering clip buffers? I'm guessing that there are fewer ops that go into producing a clip buffer. But, are you also marking other buffers permanent as well?
11-06-2015
changeset: http://hg.openjdk.java.net/openjfx/9-dev/rt/rev/0fc4afb1c047
11-06-2015
Kevin, Jim, thanks for the review and comments. Jim, yes the problem was already mentioned here: RT-25344. So, thanks for another clear description. "The reason the Canvas backbuffer is marked permanent is that we have absolutely no way to reconstruct it since the commands used to render it were lost after we ran them. We plan to eventually introduce API for developers to listen for texture-loss events and repair them, but until we have such an API, we mark them permanent." Well, for WebView html5 canvas there's hypothetically a second option: keep the render ops queue. It should be technically possible (we anyway manage the queue for WebView), still it requires thorough investigation of pros and cons.
11-06-2015
I have 2 general comments which fall under "we're not going to solve them here and now", but I want to increase the awareness of these concerns and make sure that people aren't getting too comfortable with these solutions in the long run: - We should be looking for any way to avoid making resources permanent. In most cases we should look at the permanent setting as a short term work around for an eventual rearchitecture of the code in question to be able to respond to lost surfaces. - We should not be using internal Decora interfaces to manage cached textures. The caching/reuse there is for managing arbitrary chains of effects where the result gets used right after the effect is done processing and then returned. Unfortunately we have no alternative "cache/reuse" mechanism for RTTs presently so we end up with lots of code digging into this internal code in Decora to achieve its ends. Sigh. One problem remains with this fix that we had before, but was probably never noticed... Textures marked permanent are not permanent. (Surprise!). Instead, the permanent flag has 2 effects: - It prevents our resource management code from deleting them when space is low - It tells our resource management to not worry about the lock/unlock counts But, it does not prevent the system from deleting the textures. ES2 doesn't do this, but D3D can (I believe when we get some sort of device lost event?). So, making them permanent is no guard against having the texture go away - it just makes them not go away frivolously. Note that the existing code already has this issue since regular textures can also be deleted by the system under the same circumstances. So, we didn't cause any new problems here, but let us not convince ourselves that this solution is complete. It's better than it was, though, and that is an important consideration for now. Having said all of that, I think the proposed solution is a good minimal change that gets rid of the resource locked issues, appears to not abuse resources too badly according to the testing I see mentioned above, and doesn't cause any new bugs, though it fails to fix an existing bug as per my previous paragraph. +1 from me, but we need to plan to: - Deal with these textures going away (outside of our control) - If we can deal with that, then we can make them non-permanent and reconstruct them as necessary, but we'd need to balance the locks - At some point I plan to create a more general rtt cache/reuse mechanism which may make code like this easier without having to muck around with PrDrawables
11-06-2015
Version 2 looks good to me, and seems like the right way to manage permanent resources. So unless Jim has any concerns, I'm fine with the proposed fix. My testing looks good, too. +1
10-06-2015
[added Jim as a watcher] I will review and test it this afternoon.
10-06-2015
Kevin, I addressed your suggestion. PrDrawable is now created with a dedicated RTT in case of "permanent" layer. Checked with the scenario of multiple reloading of a canvas. The number of permanent textures is balanced around the initial value, not growing up. Also checked with the w3c samples: http://www.w3schools.com/html/html5_canvas.asp. webrev: http://cr.openjdk.java.net/~ant/RT-46160/webrev.2
10-06-2015
I chatted with Anton about this offline, but wanted to summarize here. Using a PrDrawable texture obtained from the ImagePool via createCompatibleImage, and then marking it as permanent seems like the wrong thing to do. Especially if it is ever returned to the pool. A better approach might be to create such a permanent texture as an RTT and then wrap it in a PrDrawable if it needs to be used in an effect.
10-06-2015
I agree, my assumption about Java up calls was unjustified. I can't reproduce the crash, so I can only try to figure it out indirectly. The crash happened in a C frame, at: env->CallVoidMethod(getWCRenderingQueue(), midFwkDisposeGraphics); I looked into jni.cpp for the CallVoidMethod implementation and found out it throws NPE when the obj (getWCRenderingQueue()) is null. If methodID is 0 it can crash, however the stack looks different (jni_CallVoidMethod is at the top). Also, midFwkDisposeGraphics can not be 0 in the dtor as a static var. So, the only possibility is null env, in which case the stack looks just like that. Also, I see no Java frame in the hs dump and the following: VM state:at safepoint (shutting down) That is, your assumption about null env due to VM detach is likely correct. (I checked the spec and GetEnv indeed returns NULL and JNI_EDETACHED err code in this case). So to fix the crash we should only check for env != NULL and otherwise silently return. Thanks for your hint! Additionally, I tested the scenario you suggested: reloading the canvas html multiple times. I can observe permanent texture growing from 6 up to 11 but then it stops at that level. And what I found out is interesting. We use PrDrawable for a Layer buffer. It's retrieved with Effect.getCompatibleImage and freed with Effect.releaseCompatibleImage. The pair of methods check out and then check in an image from a pool. An image is locked (an underlying texture is locked) and then unlocked. In case it's not unlocked during a render pass, an "outstanding resource locks detected" message is printed (as before the fix). So, I made the underlying texture permanent to avoid that. However, when PrDrawable is released its texture is not changed (except that it is unlocked) and it remains permanent. This is the reason of growing permanent textures in the prism pool stats. Also, the same PrDrawable can be checked out from the pool on a subsequent request (in case it matches the requested size). This is what likely happens with the test case. At some point the same PrDrawable instances are reused (canvas size doesn't change). So, my question is how PrDrawable is supposed to be used as a permanent resource? I can't make a texture non-permanent on releasing PrDrawble, I can only dispose it, in which case a subsequent getCompatibleImage will forward to createCompatibleImage as it will find a texture invalid. Is this a correct way of using PrDrawable? webrev: http://cr.openjdk.java.net/~ant/RT-46160/webrev.1
10-06-2015
I don't see how the addRenderJob could cause this particular crash, and I haven't ever seen this sort of problem with it. Based on the crash dump it looks like it's coming from the native method. One other thing that could cause this is if WebCore_GetJavaEnv() returned null. If it was called after the Java VM was detached I could imagine that happening. Hard to say without being able to reproduce it. Stack: [0x0000000103231000,0x00000001032b1000], sp=0x00000001032b0c70, free space=511k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [libjfxwebkit.dylib+0x48e6ad] _ZN7JNIEnv_14CallVoidMethodEP8_jobjectP10_jmethodIDz+0x7d C [libjfxwebkit.dylib+0xa52e6a] _ZN7WebCore14RenderingQueue15disposeGraphicsEv+0xda C [libjfxwebkit.dylib+0xa4ed82] _ZN7WebCore14RenderingQueueD2Ev+0x12 C [libjfxwebkit.dylib+0xa4f919] _ZN7WebCore15GraphicsContext15platformDestroyEv+0x39 C [libjfxwebkit.dylib+0xa2ec03] _ZN7WebCore15GraphicsContextD2Ev+0x13 C [libjfxwebkit.dylib+0xa4df4b] _ZN7WebCore11ImageBufferD2Ev+0x1b C [libjfxwebkit.dylib+0xcbabd6] _ZN3WTF6OwnPtrIN7WebCore11ImageBufferEED1Ev+0x16 C [libsystem_c.dylib+0x5d7a1] __cxa_finalize+0xb1 C [libsystem_c.dylib+0x5da4c] exit+0x16 C [java+0x6129] CreateExecutionEnvironment+0x0 C [libsystem_pthread.dylib+0x1899] _pthread_body+0x8a C [libsystem_pthread.dylib+0x172a] _pthread_struct_init+0x0 C [libsystem_pthread.dylib+0x5fc9] thread_start+0xd
10-06-2015
env->CallVoidMethod(getWCRenderingQueue(), midFwkDisposeGraphics); Here only null getWCRenderingQueue() value may cause the crash, but I don't see how it can turn to be null. Another source of the crash is the Java code: Toolkit.getToolkit().addRenderJob(new RenderJob(r)); May it fail on exit? Or may the render job fail on exit? Did you face with anything similar in Glass?
09-06-2015
Oh, thanks for the catch! I'll investigate it then...
09-06-2015
I only saw that crash on exit failure once (on my mac) so it's very intermittent.
09-06-2015
I had just given a "+1" but found a crash on exit in dispose().
09-06-2015
After running my test for a while on Mac I then exited WebLauncher and got the crash I indicated above. The hs_err crash log is attached.
09-06-2015
I was able to get to to hit the disposal code. So it looks OK. One last thing to check in the code and then I'll be done.
09-06-2015
I see. It would be good to test that this actually is working. In the mean time, I'll finish the review now.
09-06-2015
Kevin, thanks for your testing. Canvas is backed by an image buffer implemented in ImageBufferJava.cpp. In its ctor it calls for WCRenderQueue.createBufferedContextRQ() and passes a returned JNI object to PlatformContextJava passed to GraphicsContext. The latter is implemented as GraphicsContextJava. And the chain of references to the original RQ object becomes this: GraphicsContextPlatformPrivate* m_data (a raw ptr declared in GraphicsContext.h, casted from PlatformGraphicsContext, a typedef for PlatformContextJava), and RefPtr<RenderingQueue> m_rq (a ref ptr declared in PlatformContextJava.h). Then, m_data is deleted from GraphicsContext::platfromDestroy() called from GraphicsContext dtor. The GraphicsContext object (associated with ImageBuffer) is declared as OwnPtr<GraphicsContext> m_context in ImageBuffer.h (a smart ptr w/o ref counting). So, its destructed when the ImageBuffer goes away, causing the chain mentioned above to be destructed as well, causing the RenderingQueue::disposeGraphics() (introduced in the fix) to be called. The ImageBuffer itself is managed by WebKit. I can't say exactly what is its life cycle, but I suppose it gets destructed by JS GC in case of Canvas. So, what I'm trying to say is that the disposal should be triggered by WebKit and it ends up on the Java side where we release layer's resources. Sure, I can't guarantee there are no bugs in this chain, but at least it's not evident as for now...
09-06-2015
If I run with "java -Dprism.poolstats=true" and continually reload the clip-test.html page (pressing the "Go" button repeatedly) I can see the texture memory continuing to grow. It also does this before the fix, but what this means is that the fix has made the warning message go away, but it doesn't seem that it has fixed the underlying resource consumption problem.
09-06-2015
As part of my review, I'm running tests. It does resolve the issue with the test case, but I found one possible issue. Calling makePermanent will make the outstanding resource lock issue go away, but unless those textures are eventually disposed we will have a resource leak. This seems to be the case in my testing. If I run with -Dprism.poolstats=true I can see the number of permanent texture grow as I navigate to various websites, and they don't seem to be released. I can't tell if it is growing indefinitely...it doesn't appear to be, so maybe this is OK. I will finish my review, but wanted to get this feedback out there so you could take a look.
09-06-2015
Having that said, a solution which naturally comes to mind is to mark the texture permanent when it relates to a persistent context. There're two types of layers in WebView: transparency and clip. The former though, as far as I understand it, should always go in the pair - beginTransparencyLayer/endTransparencyLayer - not depending on a context type. The methods are defined on WCPrismGraphicsContextClass and they take care of the layer disposal. So, the only layer which may not be explicitly disposed is the clip layer. Also, in the discussion in RT-38290 it was said that a canvas context (WCBufferedContext) is disposed when its native peer is destructed when the canvas is gone and reclaimed by the JavaScript GC. Actually, this doesn't seem to be the case. The native peer is destructed indeed, but the java WCBufferedContext instance is silently released without calling dispose() on it. Here's the fix which does two things: 1. Marks PrDrawable texture permanent in appropriate context. 2. Explicitly calls WCBufferedContext.dispose() on its native peer destruction. The 2nd is safe in case of multiple invocations (in case dispose() is called first from java and then from native). webrev: http://cr.openjdk.java.net/~ant/RT-46160/webrev.0
03-06-2015
The leaked resource can be detected with -Dprism.pooldebug=true (along with setting ManagedResource.trackLockSources=true in the source code). Here's what I got: [java] Outstanding resource locks detected: [java] ES2 Vram Pool: 27,254,784 used (5.1%), 67,108,864 target (12.5%), 536,870,912 max [java] java.lang.Throwable: 1 [java] at com.sun.prism.impl.ManagedResource.lock(ManagedResource.java:143) [java] at com.sun.prism.impl.ManagedResource.<init>(ManagedResource.java:62) [java] at com.sun.prism.impl.DisposerManagedResource.<init>(DisposerManagedResource.java:34) [java] at com.sun.prism.es2.ES2TextureResource.<init>(ES2TextureResource.java:34) [java] at com.sun.prism.es2.ES2RTTexture.create(ES2RTTexture.java:290) [java] at com.sun.prism.es2.ES2ResourceFactory.createRTTexture(ES2ResourceFactory.java:158) [java] at com.sun.prism.es2.ES2ResourceFactory.createRTTexture(ES2ResourceFactory.java:154) [java] at com.sun.scenario.effect.impl.prism.ps.PPSDrawable.create(PPSDrawable.java:59) [java] at com.sun.scenario.effect.impl.prism.ps.PPSRenderer.createCompatibleImage(PPSRenderer.java:163) [java] at com.sun.scenario.effect.impl.prism.ps.PPSRenderer.createCompatibleImage(PPSRenderer.java:67) [java] at com.sun.scenario.effect.impl.ImagePool.checkOut(ImagePool.java:183) [java] at com.sun.scenario.effect.impl.Renderer.getCompatibleImage(Renderer.java:116) [java] at com.sun.scenario.effect.impl.prism.ps.PPSRenderer.getCompatibleImage(PPSRenderer.java:168) [java] at com.sun.scenario.effect.impl.prism.ps.PPSRenderer.getCompatibleImage(PPSRenderer.java:67) [java] at com.sun.scenario.effect.Effect.getCompatibleImage(Effect.java:479) [java] at com.sun.javafx.webkit.prism.WCGraphicsPrismContext$Layer.<init>(WCGraphicsPrismContext.java:1335) [java] at com.sun.javafx.webkit.prism.WCGraphicsPrismContext$ClipLayer.<init>(WCGraphicsPrismContext.java:1401) [java] at com.sun.javafx.webkit.prism.WCGraphicsPrismContext$ClipLayer.<init>(WCGraphicsPrismContext.java:1394) [java] at com.sun.javafx.webkit.prism.WCGraphicsPrismContext.setClip(WCGraphicsPrismContext.java:325) [java] at com.sun.webkit.graphics.GraphicsDecoder.decode(GraphicsDecoder.java:222) [java] at com.sun.webkit.graphics.WCRenderQueue.decode(WCRenderQueue.java:91) [java] at com.sun.webkit.graphics.WCRenderQueue.decode(WCRenderQueue.java:102) [java] at com.sun.webkit.graphics.WCRenderQueue.decode(WCRenderQueue.java:109) [java] at com.sun.webkit.graphics.GraphicsDecoder.decode(GraphicsDecoder.java:339) [java] at com.sun.webkit.graphics.WCRenderQueue.decode(WCRenderQueue.java:91) [java] at com.sun.webkit.WebPage.paint2GC(WebPage.java:689) [java] at com.sun.webkit.WebPage.paint(WebPage.java:656) [java] at com.sun.javafx.sg.prism.web.NGWebView.renderContent(NGWebView.java:96) [java] at com.sun.javafx.sg.prism.NGNode.doRender(NGNode.java:2053) [java] at com.sun.javafx.sg.prism.NGNode.render(NGNode.java:1945) [java] at com.sun.javafx.sg.prism.NGGroup.renderContent(NGGroup.java:235) [java] at com.sun.javafx.sg.prism.NGRegion.renderContent(NGRegion.java:576) [java] at com.sun.javafx.sg.prism.NGNode.doRender(NGNode.java:2053) [java] at com.sun.javafx.sg.prism.NGNode.render(NGNode.java:1945) [java] at com.sun.javafx.tk.quantum.ViewPainter.doPaint(ViewPainter.java:477) [java] at com.sun.javafx.tk.quantum.ViewPainter.paintImpl(ViewPainter.java:330) [java] at com.sun.javafx.tk.quantum.PresentingPainter.run(PresentingPainter.java:91) [java] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [java] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [java] at com.sun.javafx.tk.RenderJob.run(RenderJob.java:58) [java] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [java] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [java] at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:125) [java] at java.lang.Thread.run(Thread.java:745) [java] 10 total resources being managed [java] average resource age is 0.5 frames [java] 0 resources at maximum supported age (0.0%) [java] 5 resources marked permanent (50.0%) [java] 1 resources have had mismatched locks (10.0%) [java] 1 resources locked (10.0%) [java] 6 resources contain interesting data (60.0%) [java] 0 resources disappeared (0.0%) So, the problem is that ClipLayer created for WCBufferedContext (which backs the canvas) is not disposed and the PrDrawable texture it creates is not released. Usually WCGraphicsPrismContext (a super class of WCBufferedContext) is disposed at the end of a render pass, however this is not the case for WCBufferedContext. The latter may back a persistent render target (just like canvas) and its life cycle is managed from the native side and it may survive potentially unlimited number of render passes. Some related discussion on canvas/clip and graphics context in WebView can be found in RT-38290.
03-06-2015