Bug ID: JDK-8092801 [Canvas] Canvas buffers commands indefinitely when it isn't visible leading to Out of Memory Exception

JDK 8
8u20Fixed

Thanks Jim! If possible, could we make Zoomy available as a test case too?
28-04-2014
Fixed in the 8u-dev repo with the following changeset: changeset: 6878:8979d8ffbc06 date: Fri Apr 25 16:44:09 2014 -0700 summary: Fix RT-24903: Canvas buffers rendering commands without consuming them if not visible. http://hg.openjdk.java.net/openjfx/8u-dev/rt/rev/8979d8ffbc06 Tested manually with the reduced test case submitted above in CanvasLeak.zip.
25-04-2014
Looks great. +1
25-04-2014
renderForcedContent() wins in the polls. New webrev: http://cr.openjdk.java.net/~flar/RT-24903/webrev.10/ - No more debugging prints - I left the paintImpl(null) as is to avoid having to break out the root rendering capability into a duplicate function - NODE_FORCE_SYNC and renderForcedContent() are the names I went with.
25-04-2014
Requisite? Essentials? renderBasal()? renderIntrinsics()? renderHeartbeat()? renderPing()? renderRecurring()? renderPeriodic()? renderAlways()? And renderForcedContent() isn't so bad. For one thing, it applies to more things than just periodic renderings. Also, renderRequiredContent() is in the same vein, but implies need rather than obligation?
25-04-2014
It's kind of contingency-ish, but I don't like that word because it implies "future" and "possibilities". I looked up synonyms and the online thesaurus listed "if it's cool", too bad you can't use apostrophes in method names. ;)
25-04-2014
It's a tough one. How about renderForcedContent() to match NODE_FORCE? Not sure I like that much either.
25-04-2014
That might work. You would want to document it as rendering only those non-visible nodes that "care" in some way.
24-04-2014
Even "draining" is only one reason why a node might need to be visited. It's a render-time operation so I think renderFoo() is a reasonable prefix, perhaps renderNonVisible()?
24-04-2014
I guess you want a name that suggests "render nodes that need to be processed (drained) even when not doing a normal rendering pass". So maybe one of the following? drainNodes() renderNodesForDraining() renderSpecialNodes() Not sure I like any of those, but maybe they will suggest something else.
24-04-2014
It would be a good task for someone to modify the visibility changes so that they also use NODE_FORCE so we only have one dirty bit to check as an exception to synchronization. I can change it to FORCE_SYNC, that might be a nicer name, but I hesitate to modify the way that visibility dirty bits are managed. The thing I don't like about renderCanvases() is that it is specific to Canvas, but we may discover other reasons to need to visit nodes during rendering that have nothing to do with the current Canvas issue. We really only need one mechanism for anything that has bookkeeping to be done even when it is not in a dirty rect and the Canvas is just the first object that can use that. I'm seeing this as the NG layer equivalent to the new generically-named NODE_FORCE_SYNC flag I added at the app layer. Originally I thought that there was more stuff in paintImpl() that we would be benefitting from, but I guess not. "root" is not exposed to the subclasses which is silly and complicates going directly to renderCanvases(). I suppose I could make it protected? On the other hand, in paintImpl() we still will want to call root.renderCanvases(g) when the dimensions are zero, so that code will still remain for that case - and I thought I could releverage it by passing in a null for the case of not being visible. If I call renderCanvases() directly from PP/UP then I won't need the null graphics test at the top of paintImpl(), but I will still need to call renderCanvases() there for the 0 dimensions case. validateStageGraphics() seems to mostly fail on !visible or having zero dimensions, both of which we want to still render Canvases during - it also looks like it can fail for window==null or view==null, neither of which would necessarily prevent us from rendering Canvases, but I don't know all the conditions during which we might fail - Kevin?
24-04-2014
The v09 version of the fix looks good to me. * I like renderCanvas() as a name. NODE_FORCE seems OK too (or maybe NODE_FORCE_SYNC?) * I don't have a strong opinion about paintimpl(null) versus paintCanvases(), but the meaning of null should be documented if you stick with the former.
24-04-2014
I like this code much better that the previous webrev. We can address the problem of canvases "floating in space" in another JIRA. 1) I like renderCanvas() because it fits in with the other named rendering methods. 2) Can I ask about paintImp(null)? Why not just have a new method "paintCanvases() and call that? That would be explicit rather than looking for null. 3) Are there circumstances where validateStageGraphics() will fail (not because we are iconified etc.) and we should not attempt to paint?
24-04-2014
New webrev with the Disposer fix, just a small change to the tail end of Presenting/UploadingPainter files: http://cr.openjdk.java.net/~flar/RT-24903/webrev.09/ This was able to run zoomy iconified for several minutes with no drop in framerate on Windows and Mac. I'm still not entirely happy with the name "paintCanvases" - suggestions? paintContingencies()? paintBackground()? paintForced()? Also, I'm curious why one call to Disposer.cleanUp() was inside the lock of the back buffer and the other was outside (comparing PresentingPainter and UploadingPainter). Any ideas?
24-04-2014
Ah, one problem might be that we only call Disposer.cleanUp() if valid is true, but in the case of an iconified window, valid is false and we call the backup "paintImpl(null)" then transfer to the finally where we skip the Disposer.cleanUp(). The fix could be to eliminate the test on Disposer.cleanUp(). Is there a reason we don't try to cleanup? It doesn't look like it will take much time if we aren't rendering anything.
24-04-2014
Yes, that is running out of texture memory. It looks like it is only accumulating 2 frames there, was it going through periods of accumulating more frames than that? Zoomy is particularly egregious in terms of per-frame texture usage since each frame has a 4x32-bit per pixel float texture that gets created and dropped on the floor. Still, unless we are leaking and holding on to them, they should be deleted when we need more space...? Does that happen if you leave it open for a similar amount of time? Also poolstat logs might help track any resource leaks...
23-04-2014
A quick review of the code changes in the latest version look good to me. I still need to do a detailed review, but I still see one problem on Windows. When iconifying the original unmodified Zoomy, it runs for quite a while, but then gets this error in rendering repeatedly and eventually runs OOM after that. Maybe running out of texture memory? java.lang.IllegalArgumentException: Texture must be non-null at com.sun.scenario.effect.impl.prism.PrTexture.<init>(PrTexture.java:38) at com.sun.scenario.effect.impl.prism.ps.PPSRenderer.createFloatTexture(PPSRenderer.java:182) at com.sun.scenario.effect.FloatMap.getAccelData(FloatMap.java:221) at com.sun.scenario.effect.impl.prism.ps.PPSTwoSamplerPeer.filterImpl(PPSTwoSamplerPeer.java:105) at com.sun.scenario.effect.impl.prism.ps.PPSEffectPeer.filter(PPSEffectPeer.java:54) at com.sun.scenario.effect.CoreEffect.filterImageDatas(CoreEffect.java:106) at com.sun.scenario.effect.DisplacementMap.filterImageDatas(DisplacementMap.java:328) at com.sun.scenario.effect.FilterEffect.filter(FilterEffect.java:195) at com.sun.javafx.sg.prism.NGCanvas.applyEffectOnAintoC(NGCanvas.java:659) at com.sun.javafx.sg.prism.NGCanvas.renderStream(NGCanvas.java:938) at com.sun.javafx.sg.prism.NGCanvas.renderCanvases(NGCanvas.java:600) at com.sun.javafx.sg.prism.NGGroup.renderCanvases(NGGroup.java:198) at com.sun.javafx.tk.quantum.ViewPainter.paintImpl(ViewPainter.java:168) at com.sun.javafx.tk.quantum.PresentingPainter.run(PresentingPainter.java:58) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at com.sun.javafx.tk.RenderJob.run(RenderJob.java:58) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:125) at java.lang.Thread.run(Thread.java:744) syncing 200 commands and 6 objects accumulated 400 commands and 12 objects ... Exception in thread "JavaFX Application Thread" java.lang.OutOfMemoryError: Java heap space
23-04-2014
A fairly complete new version: http://cr.openjdk.java.net/~flar/RT-24903/webrev.08/ This one handles iconified canvases - they are both flushed and rendered - but it still does not handle orphan canvases (outside the needs/scope of this fix). It still has 2 debug printouts to show when the canvas is flushed to the NG layer and whether or not data is accumulating at the NG layer without being rendered. I'll gratefully consider better names for "NGNode.renderCanvases(...)". And DirtyBits.NODE_FORCE as well for that matter...
23-04-2014
Here is a new webrev that only tries to solve the problem for Canvas nodes that exist in a scene. Currently it does not work for: - orphan canvases are not flushed or rendered - iconified canvases are flushed, but not rendered The mechanism to ensure that all (non-iconified, non-orphan) canvases have consumed their buffer is a little heavy-handed in this webrev, it basically does a full tree visit, but actions are only taken on NGCanvas objects and only if they haven't already been consumed. Some way of marking just the branches on the tree as requiring a search for canvases, or some way of recording a list of dirty canvases in the Scene object could be more efficient: http://cr.openjdk.java.net/~flar/RT-24903/webrev.07/
23-04-2014
After discussing the locking issues in the current fix, we decided to reduce the scope of the fix for 8u20 to only deal with flushing canvas objects that are part of an existing scene. This should fix the test case for this issue just fine, but it won't solve the issues with the test cases that Kevin wrote all of which test our ability to flush canvas objects that are actively animating without being added to a scene.
22-04-2014
Latest webrev: http://cr.openjdk.java.net/~flar/RT-24903/webrev.06/ This seems to work well with the submitted test case and the two Zoomy test cases that Kevin wrote on both Mac and Windows. The following issues with the fix still remain: - remove printouts (there for checking progress for now) - remove direct references to renderLock from Canvas and NGCanvas - Is the solution for the disposed swap chain optimal? (returns true from ES2SwapChain.lockResources and marks the scene for repaint in PresentingPainter) - Note that I also mark the Swap Chain back buffer as "contentsUseful()" if dirty opts is enabled so that we are less likely to free a back buffer when looking for dead texture memory.
19-04-2014
I made a temporary simple solution for the repaint with disposed SwapChain issue by marking the presentable for recreation when it discovers its swap chain was disposed and then requesting the entire scene to repaint whenever the presentable is marked for recreation.
19-04-2014
I have tracked down why the Mac ES2 window goes black for Zoomy2. Zoomy creates a lot of textures and drops them on the floor, this eventually leads us to clean up old textures in order to proceed. When that happens, the algorithms for cleaning up end up disposing the back buffer of the swap chain along with all of the unused textures as it is not (currently) sophisticated enough to notice that the swap chain has been used more recently than the scratch textures. We could make it more sophisticated, but the point is that it is possible for that swap chain texture to be deleted by asynchronous means (if we have more than one window active and more than one swap chain, then the actions of one window may deplete vram outside of the scope where the swap chain for the other window would be locked and it would be prone to being freed. The problem comes in that the actions of updating the scene on the next pulse discover that the texture disappeared and recreate it, but they do not mark the scene as being "fully dirty, repaint everything please". A lot of conditions can cause us to mark a scene as being fully dirty and needing a complete repaint, but this one does not. Worse yet, it is discovered in a part of the code that doesn't have decent access to the scene object to mark it as dirty so some work will have to be done to bridge that gap. D3D does not suffer from this issue because it marks the swap chain as unrecoverable in that case and the swap chain is replaced and the act of replacing the swap chain with a new one causes an entire scene repaint, but ES2 tries to be clever and keep the swap chain, but without any way to signal the need for a full repaint. The simple solution could be for the ES2 swap chain to just ask for a new swap chain to be created, but the reuse of the old swap chain object was done for performance reasons, so hopefully a less drastic solution can be found...
19-04-2014
The latest patch seems to work fine on Windows, but it still suffers from the "black window after a few refreshes" for the Zoomy2 test case on Mac: http://cr.openjdk.java.net/~flar/RT-24903/webrev.05/ I'm posting just to document progress and because the current webrev raises a few questions to answer while I debug the black window refresh issue: - I'm having to insert direct calls to the renderLock in two places where it is needed "in practice", but there is no current interface to get the lock going through normal channels. Some new API on Toolkit or Scene or somewhere central will be needed to expose this locking mechanism without breaking encapsulation of the toolkit. - The way that RenderJobs are handled is fairly fragile with a lot of presumptive code in various places that "does the right thing" based on what is known to be happening elsewhere. The RenderJob concept needs better encapsulation for robustness if no other reason and that may provide a solution to the first issue above.
18-04-2014
The threading looks fine in the new version. One more place you could lambdify is the cvRenderJob in NGCanvas. I ran the patch and I still see the same problem with iconify, though. Also, if I add nothing to the scene (or at least nothing animating) then it gets OOM almost right away.
11-04-2014
(Also, i tried to lambda-fy the new code as best I could from examining similar code in other places...)
09-04-2014
I believe I have dealt with the threading issues in this update, and found better locations/hooks from which to drive the synchronization that should cover more cases. Here is the latest webrev: http://cr.openjdk.java.net/~flar/RT-24903/webrev.04/ Note that I haven't yet tested on Windows, but this works on Mac in the following test cases: - Zoomy + don't add it to any scene (but do launch a stage with an empty scene) - Zoomy + don't add it to any scene, but launch a stage with an active scene I did notice that the second test case turns black after a few frames (and every few frames after that) which indicates that we are fighting somewhat over the state of the common OGL context. I still have to track down what is causing that, but at least the updates get through and are rendered on most every frame now.
09-04-2014
Note that this doesn't address the (now separate) issue of what happens during a screen lock on Windows (due to a D3D lost surface). We should file a separate JIRA to track this issue. Edited to add: the lost surface issue is already filed and tracked by RT-23312
07-04-2014
Yes, might work, since the only other reference to it is always on the FX app thread. The same trick won't work in NGCanvas (since it is the renderer thread that iterates them), but a possible solution there is to have a temporary list that the constructor calls (synchronized), and then copy out of the temp list into the allCanvases list and clean the temp list (also synchronized). You wouldn't have the overhead of copying the active list, only any new canvases.
03-04-2014
What about Platform.runLater(allCanvases.add(this)) to solve the threading problem for Canvas.allCanvases? That avoids copying it in the method that iterates over it which is executed once on every frame.
03-04-2014
The overall design looks good to me. I think it is OK for PaintCollector to call NGCanvas.repaint() given that that method just adds a render job. It might be worth adding a comment to repaint() indicating that it is called on the FX app thread. The other alternative would be to move NGCanvas.repaint() and cvRenderJob to QuantumToolkit (in which case you would leave forceRender public). I don't really mind one way or another, so probably just leaving it as you have it is fine. As for whether there are reasons that the various runnables might not be run, I think there is a problem if there are no active scenes (see my testing comments below). There might be for the case where the Canvas is on an active scene, but I can't think of any off-hand. I did find a couple of issues that need to be resolved. Canvas.java: * The allCanvases list is not thread safe. A Canvas can be constructed on a thread other than the FX app thread, so you will need to synchronize the access to the list, and also may need to copy the list when you iterate over it in forceSync() to avoid a ConcurrentModificationException when accessing the iterator. * Minor point: allCanvases can be private NGCanvas.java: * As above, the access to allCanvases is not thread-safe; the NGCanvas node is constructed on the FX app thread, and the forceRender is run on the PrismRenderer thread. Given that the peer is usually created in the node sync, I'm not 100% sure that concurrent access can happen in practice, but it seems best to guard against it. * You might add a comment to repaint() indicating that it is called from the app thread. * Minor point: cvRenderJob, allCanvases, and forceRender() can be private Testing: I ran several tests, and the fix looks good for Canvases that are attached to a scene, but doesn't yet quite work for Canvases not attached to a scene. I see one of two different failures: 1) If there is no active scene, an app that writes to a Canvas repeatedly will still get an OOM (e.g., a modified "Zoomy" that doesn't add the Canvas to the root of the scene). 2) If the same program has an active Scene with an animating rectangle (but the Canavs is not attached to a scene), it works fine when the window is not iconfied, but I get a strange error when the window is iconified. These aren't likely related to the threading concern I raised above since the mods to the detached Canvas are done on the FX app thread. Canvas synching 200 bytes and 6 objects Accumulating to 400 bytes and 12 objects RenderJob.run: internal exception java.lang.IllegalArgumentException: Texture must be non-null at com.sun.scenario.effect.impl.prism.PrTexture.<init>(PrTexture.java:38) at com.sun.scenario.effect.impl.prism.ps.PPSRenderer.createFloatTexture(PPSRenderer.java:182) at com.sun.scenario.effect.FloatMap.getAccelData(FloatMap.java:221) at com.sun.scenario.effect.impl.prism.ps.PPSTwoSamplerPeer.filterImpl(PPSTwoSamplerPeer.java:105) at com.sun.scenario.effect.impl.prism.ps.PPSEffectPeer.filter(PPSEffectPeer.java:54) at com.sun.scenario.effect.CoreEffect.filterImageDatas(CoreEffect.java:106) at com.sun.scenario.effect.DisplacementMap.filterImageDatas(DisplacementMap.java:328) at com.sun.scenario.effect.FilterEffect.filter(FilterEffect.java:195) at com.sun.javafx.sg.prism.NGCanvas.applyEffectOnAintoC(NGCanvas.java:691) at com.sun.javafx.sg.prism.NGCanvas.renderStream(NGCanvas.java:971) at com.sun.javafx.sg.prism.NGCanvas.consumeBuffer(NGCanvas.java:610) at com.sun.javafx.sg.prism.NGCanvas.forceRender(NGCanvas.java:183) at com.sun.javafx.sg.prism.NGCanvas$1.run(NGCanvas.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at com.sun.javafx.tk.RenderJob.run(RenderJob.java:58) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at com.sun.javafx.tk.quantum.QuantumRenderer$PipelineRunnable.run(QuantumRenderer.java:129) at java.lang.Thread.run(Thread.java:744) Outstanding resource locks detected: D3D Vram Pool: 19,968,000 used (7.4%), 19,968,000 managed (7.4%), 268,435,456 total 8 total resources being managed average resource age is 43.4 frames 0 resources at maximum supported age (0.0%) 2 resources marked permanent (25.0%) 1 resources have had mismatched locks (12.5%) 1 resources locked (12.5%) 2 resources contain interesting data (25.0%) 0 resources disappeared (0.0%) Scene: synchronizeSceneNodes Canvas synching 200 bytes and 6 objectsRenderJob.run: internal exception
03-04-2014
A new version of the fix. This version always fully synchronizes the Canvas to the NG layer on every frame so that it returns to a clean state to capture future rendering. http://cr.openjdk.java.net/~flar/RT-24903/webrev.02/ - Still has the debug printouts. - Still calls NGCanvas from PaintCollector (not ViewPainter as I mistakenly said above). - Still not sure if there are reasons that the various runnables that I've hooked into might not get run. - Now fully synchs every Canvas and clears its dirty flags on every frame.
03-04-2014
It turns out that the dirty flags are more of an issue than I originally thought. If I don't clear them then the next time we go to render and mark the node dirty, it will not get added to the list of dirty nodes on the scene which also means the scene will not be marked dirty. The flow of control is that a node marks a dirty flag on itself. If the node was previously completely clean, then it will add itself to the list of dirty nodes on the scene and mark the scene as dirty. Then on the next sync pulse, the dirty scenes are iterated, most all of their nodes are synchronized. Any remaining Canvas nodes that weren't synchronized as part of the regular synch process will then be forcibly synchronized, but only their buffer and size, possibly leaving any other dirty bits still set. If we don't clear the CONTENTS flag during a "forced synch", then the node will remain dirty and the next time we allocate a buffer for it on the next frame, it will not trigger the "put me on the dirty list and mark my scene dirty" processing in Node.impl_markDirty(). If I clear the GEOM and CONTENTS flags in the forceSync() method, then the scene will correctly flip flop between clean and dirty states even when the canvas animation is the only thing happening and it is not visible, but any other change to the Canvas node will cause it to become dirty with a different flag that isn't cleared and so we end up back in the "it's not causing the scene to be dirtied" issue again. Possible solutions are to always do a full node sync instead of just synchronizing the buffer parts so that all data is synchronized and the node can go back to being fully clean in all cases, or to manually mark the scene as dirty when a new buffer is allocated just in case it wasn't dirtied by the markDirty() processing.
03-04-2014
Here is a fix that tries to manually force all rendered Canvases to synch their buffers when dirty nodes are scanned (even if they were rejected by the "visible dirty nodes" synchronizations, or not in a current scene) and then also to manually update all unrendered NGCanvas objects after all of the dirty scenes are rendered (even if they weren't visited during the render dirtyOpts processing): http://cr.openjdk.java.net/~flar/RT-24903/webrev.01/ Issues to be considered or fixed before pushing: - Obviously the debug printing needs to be deleted - Not sure if I should be calling NGCanvas (or NG*anything) directly from ViewPainter - Not sure if there are conditions under which the hooks I've added to Scene and ViewPainter will not get called for other reasons (i.e. pulses just halt due to some condition). - Not sure if I should be clearing the dirty flags on a forced Canvas synch (i.e. one that did not come from the regular impl_update() call) or not since that may mean that the canvas won't be included in dirty regions at some future time? On the other hand, if it wasn't impl_updated() due to the regular synch process then that either means it is not in a Scene or it was not visible in a scene and so before it ever gets to the screen there will be a major shift in visibility that will place it in a dirty region anyway...?
03-04-2014
I think there was some confusion over this bug. I had thought that Canvas would still consume its buffer as long as it was on an active Scene, but apparently simply scrolling it out of view can cause it to stop consuming data. I'll look into at least fixing that part sooner, but cases like screen lock where we no longer have access to the hw to render would need new API.
26-03-2014
Please note that the workaround using clearRect(...) does not work on Java7. Resetting the buffer using Java7 requires some reflection to gain access to the GrowableDataBuffer...
21-03-2014
A possible workaround for us is to clear the whole Canvas on each frame using canvas.getGraphicsContext2D().clearRect(0, 0, canvas.getWidth(), canvas.getHeight()), since we need to redraw the whole Canvas on each new frame (new camera image each frame).
20-03-2014
Is there any work being done on this bug? It is really important for us to have this issue fixed. Preferably a long time before JDK9..
20-03-2014
It used to be that just scrolling the Canvas out of view in a ScrollPane would also cause this leak. Has that been fixed?
28-08-2013
@Jochen, in addition to John Hendrikx's suggestion of using a WriteableImage, Kevin offered some workarounds in his comment on the related RT-23312 issue (I've just copied and pasted Kevin's suggestions into this comment): The javadocs will be updated to indicate the conditions under which the buffer of graphics commands will be drained, which is basically: 1) The Canvas is attached to a Scene 2) The Scene is attached to a Window that is showing 3) The Window is not iconified 4) The System is not "screen locked" (e.g., Windows-L or CTRL-ALT-DEL) The first three conditions can easily be detected by the application. The latter cannot, so we will need to add an event to allow the application to know when the window is inactive. As a possibly workaround for the lack of a proper event for #4, you may be able to use a lost focus on the Window to indicate that you shouldn't keep drawing.
28-08-2013
Hello together. It's ridiculous that this issue could not be solved within one year. Apart from that I don't really understand why this is only a major bug. According the the priority description "Major" means "Major loss of function" and "Critical" means "Crashes, loss of data, severe memory leaks". To me an out of java.lang.OutOfMemoryError: Java heap space within a few seconds with one Gig of heap space sounds severe. And I don't thing any kind of working workaround was provided yet. So to be able to use JavaFX in production environments, please fix this issue.
28-08-2013
I've switched from Canvas to WriteableImage for my use case (displaying video using a PixelWriter, where the video is coming from an external source) as I was told on the JavaFX mailing list that WriteableImage does not do any buffering (something related to Canvas being 3D and WriteableImage being 2D). For me atleast this solves the issue, but it also indicates that Canvas is unable to handle this use case. The current state Canvas is in means it's going to be very tough to use Canvas for anything animated or frequently updated without user interaction. If this occurs off screen, or when the blanker is active or the screen is locked, you can potentially run out of memory quick or after several hours depending on the kind of updates or animation you are doing.
28-08-2013
It seems we directly call System.gc(), can release the memory.
19-08-2013
Why API changes? The API of Canvas is fine from a user perspective. However this behaviour of buffering actions until the VM runs out of memory is certainly not expected. I just ran into this issue as well while displaying 1920x1080 video on a canvas at 25 frames per second. My Canvas isn't even "invisible" -- it apparently however was delayed a short while, while switching Scene to start displaying the video. That can already be enough to cause an out of memory (a 256 MB VM will go through that amount of memory in roughly 1 second, assuming my app does not use memory for anything else).
31-07-2013
This will likely require API changes for canvas and we are past feature freeze. This should be first on the plate for update release.
16-07-2013
jvisualvm screenshot showing how a ScrollPane or TabPane makes Canvas a factory for out-of-memory errors
11-09-2012
See also: RT-23312
11-09-2012

Duplicate :	JDK-8094438 - OutOfMemory error due to GrowableDataBuffer in Canvas when window is not shown
Duplicate :	JDK-8116176 - Memory leak on Windows screen lock
Relates :	JDK-8296364 - Canvas release resources programatically
Relates :	JDK-8093972 - OutOfMemoryError after pressing Ctrl+Alt+Del or screen lock while animating a canvas