JDK-8103341 : Provide API to stream raw frames of supported codecs into JavaFX media in memory
  • Type: Enhancement
  • Component: javafx
  • Sub-Component: media
  • Affected Version: 7u40
  • Priority: P4
  • Status: Closed
  • Resolution: Duplicate
  • Submitted: 2013-10-17
  • Updated: 2015-06-16
  • Resolved: 2013-10-17
Related Reports
Duplicate :  
Description
The current version of JavaFX Media only accepts "url" as a source. You cannot simply take an in-memory byte stream and feed it into JavaFX Media for playback. Contrast it with all other other JavaFX. They all take their input in multiple ways. 

For example, there is an API to take images from file as well as to programmaticaly produce them in memory. It means that I can read from my own file format or otherwise programmatically pruduce it and have JavaFX display it. JavaFX media does not currently have this kind of capability.

*API discussion*

The API user shall be able to stream data into JavaFX media at the low lower of raw encoded audio and video packets. So, if JavaFX media supports h264 or VP6 for video and mp3 or aac for audio, it should be possible to stream raw h264 or VP6 video frames and/or raw mp3 or aac audio frames into JavaFX media for playback. 

The ability to encode/decode/transcode frames of different formats is not needed (out of scope) and is much harder due to licensing issues. 

*Rationale*

Rationale for this requiest if multifold. First of all, in proprietary enterprise applications of the kind we are writing, data typically travels in our custom propritary/non-standard streams. It whould be desirable to stream multimedia data in our applications inside our proprietary data streams, extract raw frames from our proprietary streams and send them to JavaFX media on the desktop for playback.

Second rationale is that it opens door for 3rd party developers to extend JavaFX media to support more types of containers. JavaFX becomes more flexible in the kinds of content it can consume without JavaFX media team having to worry about supporting all combinations of containers and codecs which are often non-documented and/or specific to certain products. 

See, for example, RT-33271. Suppose that there is an FLV with h264 and mp3. JavaFX media supports both h264 and mp3 and plays the video and audio perfectly if I place the corresponding raw video and audio frames into mp4 container. FLV is a trivial container format to extract the raw frames from. My code to do it is about one page long, but wrapping those raw frames it into mp4 container for JavaFX media is a very non-trivial task that requires a lot of code. So, because I cannot just provide my raw frames to JavaFX media, I'm forced to do a lot of extra work:
* Wrap my raw frames into mp4 container.
* Start http server on desktop just for the purpuse of supplying frames to JavaFX Media.

So, yes, because JavaFX Media can play http there is "an API to do it". I just start http server in the same JVM and pass http://localhost:port/file url to JavaFX Media. It is a realy ugly solution to a simple problem of playing a certain stream of audio and video frames. It should be much easier.

Comments
That's fair. I can live with MPEG_TS as a way to properly timestamp my audio and video. It is ugly (e.g. desiged for a whole different purposed) but it does not take a lot of code to create an MPEG_TS stream from raw video and audio frame. I'll watch RT-26823.
18-10-2013

Currently we don't know details. But the main idea is that your streams must be timestamped properly so that the engine (GStreamer) does all synchronization correctly. We'll come up with the details latter. Currently the main demand is to support MP3 for input streams but we'll keep in our list your requirement. Thanks for you interest.
17-10-2013

The devil is in details. Why you say "encoded H264" and "AAC audio" stream what do you mean? If that means Annex B H264 NALUs and ADTS AAC, then yes. Basically, I want to be able to feed the simplest possible byte-oriented stream that standard defines. The trick is in the synchronization between them. That will require some advanced APIs. Here is my litmus test: API should be poferful enough, so that I can write my own adapter that receives RTMP audio+video stream (that JavaFX media does not support natively) and plays it back via JavaFX media in relatively straightforward way. I have my own RTSP handler which I can contribute as a testcase/example for this new API when it appears.
17-10-2013

We plan to add audio and video inputs for the MediaInputStream later when we implement recording. So you can (in some future) just feed MediaPlayer with encoded H264 stream and AAC audio stream. Is it what you expect ?
17-10-2013

It is not a duplicate. I'm asking for ability to play _raw frames_, not an multiplexed input stream. Yes, RT-26823 is going to be an improvement, but it only alleviates my pain #2 (I will not have to start my own http server). That is the lesser of pains. I can write a http server in two pages of code as a workaround for RT-26823. Being forced to multiplex my stream into a mp4 container (that JavaFX is going to demux right away) is much, much bigger pain.
17-10-2013

It's a duplicate of RT-26823. Please track the main issue. It has a bunch of info and some discussion.
17-10-2013