United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
JDK-5037133 : Mixed mode rendering and 3D effects using Java2D and JOGL together

Details
Type:
Enhancement
Submit Date:
2004-04-23
Status:
Closed
Updated Date:
2007-01-10
Project Name:
JDK
Resolved Date:
2007-01-10
Component:
client-libs
OS:
windows_xp
Sub-Component:
2d
CPU:
x86
Priority:
P4
Resolution:
Duplicate
Affected Versions:
5.0
Fixed Versions:

Related Reports
Duplicate:

Sub Tasks

Description
Name: jl125535			Date: 04/23/2004


A DESCRIPTION OF THE REQUEST :
Make it possible for a Component to contain both stuff rendered in 3D and stuff rendered in 2D, using JOGL and the new Java 1.5 OpenGL Java2D implementation together.

Enable applications to specify 3D parameters to 2D renderings, e.g. to control a 4x4 camera matrix used by Java2D, or to render 2D controls into a texture in a 3D scene.

Make it possible to apply 3D effects to Swing components rendered in a texture.

Enable cool transitions and compositing of 2D imagery using the 3D pipeline.

Add new Shape geometry to Java2D that renders efficiently through the 3D pipeline. e.g. VertexArrays.


JUSTIFICATION :
Text in JOGL is painful.  3D geometry with lighting in Java2D is painful.
JOGL + Java2D in the same window sharing space. That's the future.

Oh, as another justification, both Microsoft and Apple are already well on the way:

http://www.microsoft.com/whdc/hwdev/tech/display/graphics-reqs.mspx
http://www.apple.com/macosx/features/quartzextreme/

The industry is merging their 2D and 3D pipes. Java needs to too.
(Incident Review ID: 255529) 
======================================================================

                                    

Comments
EVALUATION

This RFE makes a lot of disparate requests, so it may help to separate them.
Let's look at some of the specific requests in the description of this bug
report, and what we've done to address them...

> Make it possible for a Component to contain both stuff rendered in 3D and
> stuff rendered in 2D, using JOGL and the new Java 1.5 OpenGL Java2D
> implementation together.

This is essentially what JOGL's GLJPanel class offers.  As mentioned above,
in JDK 6 we added an (unsupported, internal, but still accessible and usable)
API in the OpenGL-based Java 2D pipeline that allows both JOGL and Java 2D
to render directly into a GLJPanel (Swing component) without costly
intermediate steps (that are still used when Java 2D's OGL pipeline is
not available).  For more details, consult this blog entry:
http://weblogs.java.net/blog/campbell/archive/2005/09/java2djogl_inte_1.html

For a simple tutorial on how to mix 2D and 3D content using GLJPanel,
check out this blog entry:
http://weblogs.java.net/blog/campbell/archive/2006/10/easy_2d3d_mixin.html

> Enable applications to specify 3D parameters to 2D renderings, e.g.
> to control a 4x4 camera matrix used by Java2D...

This is not something we plan to enable directly in Java 2D at this time.
However, it is possible to use GLJPanel and JOGL's TextureRenderer utility
(explained below) to render into a texture using Java 2D and then manipulate
that texture in the 3D scene using standard JOGL practices.

> or to render 2D controls into a texture in a 3D scene.

Recently we've added the com.sun.opengl.util.j2d.TextureRenderer class to
Sun's JOGL implementation:
http://download.java.net/media/jogl/builds/nightly/javadoc_public/com/sun/opengl/util/j2d/TextureRenderer.html

This class allows developers to render into a Graphics2D using standard
Java 2D calls, and internally the 2D content is synced to an OpenGL texture
object.  This makes it really easy to use Java 2D to generate 2D content
and then use JOGL to map that 2D texture anywhere in a 3D scene.  (Note that
currently this class uses a BufferedImage, and therefore software rendering,
internally to manage the 2D content.  However, we're hoping to add an alternate
codepath, basically an implementation detail, that would use VolatileImage
behind the scenes when Java 2D's OGL pipeline is enabled, which would allow
for hardware acceleration of the 2D rendering as well.  This is still in
the investigation phase.)

In summary, the TextureRenderer can be used to render custom controls (and
other 2D) content and use them in a 3D JOGL scene like any other OpenGL
texture object.

> Make it possible to apply 3D effects to Swing components rendered
> in a texture.

This is similar to the last point, except more focused on Swing components
specifically.  It should be possible today to write a custom
javax.swing.RepaintManager that redirects the painting of a Swing hierarchy
into a TextureRenderer object.  The resulting texture could then be positioned
anywhere in a 3D scene using JOGL.  This could be very cool, and is something
we could probably to the com.sun.opengl.util.j2d (or perhaps ...util.swing)
package, if someone from the JOGL community wants to give it a try.

Note that there is an older demo (written before the existence of
TextureRenderer) called XTrans in the jogl-demos project that one
could refer to as a proof-of-concept:
https://jogl-demos.dev.java.net/

> Enable cool transitions and compositing of 2D imagery using the 3D pipeline.

The OpenGL-based Java 2D pipeline is focused only on rendering 2D content
in a 2D (orthographic projection) space.  As discussed above, it is already
possible using JOGL's TextureRenderer to render dynamic Java 2D content
in a 3D scene.  In addition, one can load images from the disk/network
directly into an OpenGL texture object using the nifty
com.sun.opengl.util.texture.TextureIO classes (which use Java 2D and the
Image I/O framework behind the scenes).  In either case, the resulting
OpenGL texture can be manipulated using fragment shaders, then composited
and transformed to the destination using OpenGL.  The sky is the limit.

It should be noted that the OpenGL-based Java 2D pipeline already takes
advantage of hardware blending, transforming, and (coming soon) fragment
shaders for things like convolution/blurring.  So to some extent you can
achieve these affects simply by using standard Java 2D calls, and whenever
possible the OpenGL pipeline will accelerate these things for you.  What
I'm pointing out above is that you want to enable arbitrary complex effects
that aren't available in Java 2D, it is already possible to jump down and
use JOGL/OpenGL directly (e.g. shaders).

> Add new Shape geometry to Java2D that renders efficiently through the
> 3D pipeline. e.g. VertexArrays.

It's not likely that we will add new public API for this anytime in the
foreseeable future.  It is possible that we could beef up the implementation
of our OGL-based Java 2D pipeline to automatically take advantage of things
like vertex array and display lists for complex shapes, but this is something
we've shied away from, partially due to resource constraints, and partially
because it has limited benefit for most apps.

For developers that want maximum control of this sort of thing, I would
recommend using JOGL and OpenGL (vertex arrays/buffers) directly.  Better
to allow the developer to do exactly what they want, rather than create
some new Shape classes that have to guess their intentions.

> Text in JOGL is painful.

Happily, this is no longer the case, as of the past couple weeks.  JOGL
now has a com.sun.opengl.util.j2d.TextRenderer class that makes it extremely
easy to get high-quality, 100% correct Unicode ready text in JOGL applications
(using Java 2D's text rendering APIs and the TextRenderer class internally).

As an aside, there is also one other new JOGL/Java2D utility class that I
haven't yet mentioned: com.sun.opengl.util.j2d.Overlay.  This class makes it
really easy to add a "heads-up display" to a GLCanvas or GLJPanel, and use
Java 2D to render into it.

> 3D geometry with lighting in Java2D is painful.
> JOGL + Java2D in the same window sharing space. That's the future.

This is covered by the first discussion above about GLJPanel.

On a separate topic, in an earlier evaluation it was said that we would leave
this RFE open for the purposes of considering a more "official" bridge
API between JOGL and Java 2D's OpenGL pipeline.  Way back before any of
the JOGL/Java2D interoperability work was completed, I had always assumed
that we would just extend the existing quasi-supported JAWT interfaces to
allow, for example, access to the underlying OpenGL drawables/contexts when
the OGL-based Java 2D pipeline is enabled.  However, the "bridge" interfaces
that were added in Sun' implementation of JDK 6 did not use JAWT at all,
partially because it was much more convenient to expose the necessary
functionality at the Java level (not the native level), accessible via
reflection, and partially because there is no good way to handle the necessary
locking and threaded execution via JAWT that is required by the JOGL/Java2D
interoperability layer.

The unsupported API provided as part of 6309763 is inherently tied to
Sun's implementation of the OpenGL-based Java 2D pipeline, mainly for the
purposes of enabling some amount of communication with JOGL.  It should be
noted that there is nothing JOGL-specific in that unsupported API; if the
LWJGL community, for example, wanted to use that API, they could do so in
the same way that JOGL is using it today (again, it should be noted that
the API is still considered "experimental and unsupported", but we see no
reason why it shouldn't be available in it's current form for years to come).

The purpose of JAWT in the first place was to allow a limited amount of
interoperability between native graphical libraries/toolkits (via JNI)
and AWT/Java 2D.  Now that JSR-231 (JOGL) is available as a standard
extension to the Java platform, there is much less need to offer special
hooks to our OGL pipeline via the JAWT interfaces, especially since we were
able to find a more robust, albeit "unofficial," solution as part of 6309763.
We would much rather see developers use JOGL as a rich, cross-platform way
to get extended graphics capabilities in a Java application than to, say,
use native Direct3D or OpenGL calls via JNI.

Anyway, the point I am trying to make is that given current resource
constraints, and given the fact that it is infeasible to extend the JAWT
interfaces (as described above), we have no plans to provide an "official"
bridge API for the OpenGL-based Java 2D pipeline.  We would rather devote
those resources to other projects, like coming up with new/novel APIs
(especially as we've done on the JOGL side with TextureIO, TextureRenderer,
TextRenderer, Overlay, etc).

In conclusion, the big takeaway message here is that we've done a lot to
enable mixing in both "directions" (3D content in a 2D/Swing app, 2D
content in a 3D/JOGL app), and we've done all of this without adding any
new public API in the core JDK.  Given our limited resources, we'd like to
continue this strategy of enabling public (convenience, mixing, etc) APIs
in JOGL (and even Java3D), and making unintrusive implementation changes in
Sun's OpenGL-based Java 2D pipeline where necessary.  At this time, we prefer
this approach instead of, for example, incorporating new 3D APIs directly
into the core JDK.

I'm going to close this RFE as a duplicate of 6309763, for lack of a better
option.  At least it's better than saying "will not fix"... Clearly there
is more we can do to make 2D/3D mixing even easier in Java applications, but
in closing this RFE we are saying that further development is more likely
to take place outside the JDK (and instead in the JOGL and Java3D packages).

[Longest... evaluation... ever... My apologies.]
                                     
2007-01-10
EVALUATION

A partial, unofficial solution to this problem will be offered in Mustang; see
6309763 for more details.  Basically, that RFE provides a way for JOGL's GLJPanel
to render directly into the Swing backbuffer when the OGL-based Java 2D pipeline
is enabled.  This allows a Swing application to do some rendering in Java 2D
(with the OGL backend), then pop out to JOGL to do some 3D effects, and then
switch back to Java 2D to overlay 2D rendering on top of the 3D rendering.

The changes in 6309763 represent a good start for Mustang, but there is plenty
more work to be done in this area to allow for more seamless interoperability
between the two APIs.  Therefore I will leave this RFE open as a placeholder
for future work in this area, such as providing a more official bridge between
JOGL and Java 2D, and other mixed mode solutions.
                                     
2005-08-23



Hardware and Software, Engineered to Work Together