JDK-5037133 : Mixed mode rendering and 3D effects using Java2D and JOGL together
  • Type: Enhancement
  • Component: client-libs
  • Sub-Component: 2d
  • Affected Version: 5.0
  • Priority: P4
  • Status: Closed
  • Resolution: Duplicate
  • OS: windows_xp
  • CPU: x86
  • Submitted: 2004-04-23
  • Updated: 2007-01-10
  • Resolved: 2007-01-10
Related Reports
Duplicate :  
Description
Name: jl125535			Date: 04/23/2004


A DESCRIPTION OF THE REQUEST :
Make it possible for a Component to contain both stuff rendered in 3D and stuff rendered in 2D, using JOGL and the new Java 1.5 OpenGL Java2D implementation together.

Enable applications to specify 3D parameters to 2D renderings, e.g. to control a 4x4 camera matrix used by Java2D, or to render 2D controls into a texture in a 3D scene.

Make it possible to apply 3D effects to Swing components rendered in a texture.

Enable cool transitions and compositing of 2D imagery using the 3D pipeline.

Add new Shape geometry to Java2D that renders efficiently through the 3D pipeline. e.g. VertexArrays.


JUSTIFICATION :
Text in JOGL is painful.  3D geometry with lighting in Java2D is painful.
JOGL + Java2D in the same window sharing space. That's the future.

Oh, as another justification, both Microsoft and Apple are already well on the way:

http://www.microsoft.com/whdc/hwdev/tech/display/graphics-reqs.mspx
http://www.apple.com/macosx/features/quartzextreme/

The industry is merging their 2D and 3D pipes. Java needs to too.
(Incident Review ID: 255529) 
======================================================================

Comments
EVALUATION This RFE makes a lot of disparate requests, so it may help to separate them. Let's look at some of the specific requests in the description of this bug report, and what we've done to address them... > Make it possible for a Component to contain both stuff rendered in 3D and > stuff rendered in 2D, using JOGL and the new Java 1.5 OpenGL Java2D > implementation together. This is essentially what JOGL's GLJPanel class offers. As mentioned above, in JDK 6 we added an (unsupported, internal, but still accessible and usable) API in the OpenGL-based Java 2D pipeline that allows both JOGL and Java 2D to render directly into a GLJPanel (Swing component) without costly intermediate steps (that are still used when Java 2D's OGL pipeline is not available). For more details, consult this blog entry: http://weblogs.java.net/blog/campbell/archive/2005/09/java2djogl_inte_1.html For a simple tutorial on how to mix 2D and 3D content using GLJPanel, check out this blog entry: http://weblogs.java.net/blog/campbell/archive/2006/10/easy_2d3d_mixin.html > Enable applications to specify 3D parameters to 2D renderings, e.g. > to control a 4x4 camera matrix used by Java2D... This is not something we plan to enable directly in Java 2D at this time. However, it is possible to use GLJPanel and JOGL's TextureRenderer utility (explained below) to render into a texture using Java 2D and then manipulate that texture in the 3D scene using standard JOGL practices. > or to render 2D controls into a texture in a 3D scene. Recently we've added the com.sun.opengl.util.j2d.TextureRenderer class to Sun's JOGL implementation: http://download.java.net/media/jogl/builds/nightly/javadoc_public/com/sun/opengl/util/j2d/TextureRenderer.html This class allows developers to render into a Graphics2D using standard Java 2D calls, and internally the 2D content is synced to an OpenGL texture object. This makes it really easy to use Java 2D to generate 2D content and then use JOGL to map that 2D texture anywhere in a 3D scene. (Note that currently this class uses a BufferedImage, and therefore software rendering, internally to manage the 2D content. However, we're hoping to add an alternate codepath, basically an implementation detail, that would use VolatileImage behind the scenes when Java 2D's OGL pipeline is enabled, which would allow for hardware acceleration of the 2D rendering as well. This is still in the investigation phase.) In summary, the TextureRenderer can be used to render custom controls (and other 2D) content and use them in a 3D JOGL scene like any other OpenGL texture object. > Make it possible to apply 3D effects to Swing components rendered > in a texture. This is similar to the last point, except more focused on Swing components specifically. It should be possible today to write a custom javax.swing.RepaintManager that redirects the painting of a Swing hierarchy into a TextureRenderer object. The resulting texture could then be positioned anywhere in a 3D scene using JOGL. This could be very cool, and is something we could probably to the com.sun.opengl.util.j2d (or perhaps ...util.swing) package, if someone from the JOGL community wants to give it a try. Note that there is an older demo (written before the existence of TextureRenderer) called XTrans in the jogl-demos project that one could refer to as a proof-of-concept: https://jogl-demos.dev.java.net/ > Enable cool transitions and compositing of 2D imagery using the 3D pipeline. The OpenGL-based Java 2D pipeline is focused only on rendering 2D content in a 2D (orthographic projection) space. As discussed above, it is already possible using JOGL's TextureRenderer to render dynamic Java 2D content in a 3D scene. In addition, one can load images from the disk/network directly into an OpenGL texture object using the nifty com.sun.opengl.util.texture.TextureIO classes (which use Java 2D and the Image I/O framework behind the scenes). In either case, the resulting OpenGL texture can be manipulated using fragment shaders, then composited and transformed to the destination using OpenGL. The sky is the limit. It should be noted that the OpenGL-based Java 2D pipeline already takes advantage of hardware blending, transforming, and (coming soon) fragment shaders for things like convolution/blurring. So to some extent you can achieve these affects simply by using standard Java 2D calls, and whenever possible the OpenGL pipeline will accelerate these things for you. What I'm pointing out above is that you want to enable arbitrary complex effects that aren't available in Java 2D, it is already possible to jump down and use JOGL/OpenGL directly (e.g. shaders). > Add new Shape geometry to Java2D that renders efficiently through the > 3D pipeline. e.g. VertexArrays. It's not likely that we will add new public API for this anytime in the foreseeable future. It is possible that we could beef up the implementation of our OGL-based Java 2D pipeline to automatically take advantage of things like vertex array and display lists for complex shapes, but this is something we've shied away from, partially due to resource constraints, and partially because it has limited benefit for most apps. For developers that want maximum control of this sort of thing, I would recommend using JOGL and OpenGL (vertex arrays/buffers) directly. Better to allow the developer to do exactly what they want, rather than create some new Shape classes that have to guess their intentions. > Text in JOGL is painful. Happily, this is no longer the case, as of the past couple weeks. JOGL now has a com.sun.opengl.util.j2d.TextRenderer class that makes it extremely easy to get high-quality, 100% correct Unicode ready text in JOGL applications (using Java 2D's text rendering APIs and the TextRenderer class internally). As an aside, there is also one other new JOGL/Java2D utility class that I haven't yet mentioned: com.sun.opengl.util.j2d.Overlay. This class makes it really easy to add a "heads-up display" to a GLCanvas or GLJPanel, and use Java 2D to render into it. > 3D geometry with lighting in Java2D is painful. > JOGL + Java2D in the same window sharing space. That's the future. This is covered by the first discussion above about GLJPanel. On a separate topic, in an earlier evaluation it was said that we would leave this RFE open for the purposes of considering a more "official" bridge API between JOGL and Java 2D's OpenGL pipeline. Way back before any of the JOGL/Java2D interoperability work was completed, I had always assumed that we would just extend the existing quasi-supported JAWT interfaces to allow, for example, access to the underlying OpenGL drawables/contexts when the OGL-based Java 2D pipeline is enabled. However, the "bridge" interfaces that were added in Sun' implementation of JDK 6 did not use JAWT at all, partially because it was much more convenient to expose the necessary functionality at the Java level (not the native level), accessible via reflection, and partially because there is no good way to handle the necessary locking and threaded execution via JAWT that is required by the JOGL/Java2D interoperability layer. The unsupported API provided as part of 6309763 is inherently tied to Sun's implementation of the OpenGL-based Java 2D pipeline, mainly for the purposes of enabling some amount of communication with JOGL. It should be noted that there is nothing JOGL-specific in that unsupported API; if the LWJGL community, for example, wanted to use that API, they could do so in the same way that JOGL is using it today (again, it should be noted that the API is still considered "experimental and unsupported", but we see no reason why it shouldn't be available in it's current form for years to come). The purpose of JAWT in the first place was to allow a limited amount of interoperability between native graphical libraries/toolkits (via JNI) and AWT/Java 2D. Now that JSR-231 (JOGL) is available as a standard extension to the Java platform, there is much less need to offer special hooks to our OGL pipeline via the JAWT interfaces, especially since we were able to find a more robust, albeit "unofficial," solution as part of 6309763. We would much rather see developers use JOGL as a rich, cross-platform way to get extended graphics capabilities in a Java application than to, say, use native Direct3D or OpenGL calls via JNI. Anyway, the point I am trying to make is that given current resource constraints, and given the fact that it is infeasible to extend the JAWT interfaces (as described above), we have no plans to provide an "official" bridge API for the OpenGL-based Java 2D pipeline. We would rather devote those resources to other projects, like coming up with new/novel APIs (especially as we've done on the JOGL side with TextureIO, TextureRenderer, TextRenderer, Overlay, etc). In conclusion, the big takeaway message here is that we've done a lot to enable mixing in both "directions" (3D content in a 2D/Swing app, 2D content in a 3D/JOGL app), and we've done all of this without adding any new public API in the core JDK. Given our limited resources, we'd like to continue this strategy of enabling public (convenience, mixing, etc) APIs in JOGL (and even Java3D), and making unintrusive implementation changes in Sun's OpenGL-based Java 2D pipeline where necessary. At this time, we prefer this approach instead of, for example, incorporating new 3D APIs directly into the core JDK. I'm going to close this RFE as a duplicate of 6309763, for lack of a better option. At least it's better than saying "will not fix"... Clearly there is more we can do to make 2D/3D mixing even easier in Java applications, but in closing this RFE we are saying that further development is more likely to take place outside the JDK (and instead in the JOGL and Java3D packages). [Longest... evaluation... ever... My apologies.]
10-01-2007

EVALUATION A partial, unofficial solution to this problem will be offered in Mustang; see 6309763 for more details. Basically, that RFE provides a way for JOGL's GLJPanel to render directly into the Swing backbuffer when the OGL-based Java 2D pipeline is enabled. This allows a Swing application to do some rendering in Java 2D (with the OGL backend), then pop out to JOGL to do some 3D effects, and then switch back to Java 2D to overlay 2D rendering on top of the 3D rendering. The changes in 6309763 represent a good start for Mustang, but there is plenty more work to be done in this area to allow for more seamless interoperability between the two APIs. Therefore I will leave this RFE open as a placeholder for future work in this area, such as providing a more official bridge between JOGL and Java 2D, and other mixed mode solutions.
23-08-2005