JDK-8091755 : Media should support InputStream
  • Type: Enhancement
  • Component: javafx
  • Sub-Component: media
  • Affected Version: fx2.0.2
  • Priority: P3
  • Status: Open
  • Resolution: Unresolved
  • Submitted: 2012-12-08
  • Updated: 2018-09-05
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
tbdUnresolved
Related Reports
Duplicate :  
JDK-8088930 - Inconsistent API between Media and Image class
Duplicate :  
JDK-8090627 - Allow Media objects to be loaded from all protocols.
Duplicate :  
JDK-8091053 - Media class to support other URL schemes than http, file or jar.
Duplicate :  
JDK-8103341 - Provide API to stream raw frames of supported codecs into JavaFX media in memory
Duplicate :  
JDK-8091696 - MediaPlayer doesn't play streamed mp3
Duplicate :  
JDK-8091835 - Support for shoutcast/icecast streaming (mp3/aac(+))
Duplicate :  
JDK-8092090 - aac/aac+ radio streaming
Duplicate :  
JDK-8091063 - Media playback needs to be extensible. User defined codecs or media sources should be supported.
Relates :  
JDK-8092334 - add InputStream constructor for AudioClip
Description
Currently, we can construct media only from a URL or path, but if for example I need to deencrypt an mp3 file and then play it, I have to extract it to a temporary location, losing important security and preformance, and then pass it to the Media's string constructor.
See http://stackoverflow.com/questions/13539814/playing-mp3-files-in-javafx-from-input-stream. (Another person with the same exact need and problem).

I suggest to include an InputStream constructor for media.
Comments
Bumping priority to P3 as this would help address common feature requests for supporting other streaming formats, such as RTSP or AAC streaming for internet radio. While it doesn't implement any specific streaming type, it would enable developers to create their own or allow the community to fill the gap with an implementation.
06-04-2017
I propose we consider this for JDK 10, but we will need to address the multiple (NON-muxed) stream issue, possibly by using some means other than a constructor. For example, we could implement the default constructor and add methods to add streams, once all streams have been added then another method is called to tell the player to prepare for playback, at which point it will start behaving as any other media player. Calling these new methods on a player created with a source URL would be an error.
06-04-2017
I've used wikipedia page plus a sample stream to play with. I needed a perfect solution that works for long time, so playing two at the same time was not my option. The page from wikipedia was good enough to write a code that splits mpeg ts into into packets down to actual h264 raw data. I've quickly figured the this is a dead-end path for me, so I did not actually try to do the reverse process. However, from what I saw going inside mpeg ts, it is a pretty straightforward process.
07-05-2014
If MP3 works with audio-only HLS stream... why didn't you simply have 2 media views / players, one for video and one for audio? I am running 3 simultaneously and it is working quite well... I mentioned above that I am using the Java Process class to run ffmpeg and pass it my h264 data to get mpeg-ts data and you said that it doesn't take much work to convert h264 to mpeg-ts. However, the best resource on the mpeg-ts format seems to be Wikipedia and I haven't been able to find any coding example on converting anything (let alone h264) to mpeg-ts. Do you happen to know of any resources I can use as a reference? It would improve my design by doing that myself instead of incurring the overhead and possible concurrency issues (not to mention the dependency of requiring ffmpeg to be installed) of calling ffmpeg using the Process class and re-directing the output, thanks.
07-05-2014
If you can use HLS, then go for it. That is really the best and the easiest solution. HLS is pretty straingtforward and you can write your own simple code to convert h264 into mpeg-ts stream. That does not take much work. I could not do this, unfortunately, because I had video with MP3-encoded sound which, unfortunately and for unspecified reasons (to make my life harder?), is not supported by JavaFX in HLS stream with video (even though it does support audio-only HLS stream!). So, my only choice was MP4, which is an abomination of a format. I've just used FragmentedMP4Builder sample from mp4Parser project to convert my h264 video stream and audio into a fragmented mp4 file. It works like a charm if I have a file to start with. But there is no support for streaming conversion.
06-05-2014
Yeah, the solution I ended up going with was to use the HttpServer and use Http Live Streaming (Video on Demand, i.e. the m3u8 file does not change). Also, instead of mp4parser, I discovered that I can just use ffmpeg wrapped by the Java Process class and pass the arguments to ffmpeg to use pipes (stdin and stdout) and with the Java Process, re-direct the outputs to streams instead. This was useful to get my video data (h264) into the appropriate HLS format (mpeg-ts), since I could not find any Java libraries that would perform this operation. During my research for finding a solution, I found fragmented MP4 mentioned in a few places and in the mp4parser project, I noticed the fragmentedMp4Builder class too. However, I am still unsure of how to use the mp4parser fragmented mp4 functionality and in general, how it differs from what my solution is today. Could you please provide some more detail around how you "successfully converted my stream into fragmented mp4 format that can be played back by JavaFX"? If it is another potential option for my scenario, I'd like to consider it against what I have today, thanks.
05-05-2014
I've used the same mp4Parser project. I even contributed MP3 support for mp4Parser project. It does support fragmented mp4 and I've successfully converted my stream into fragmented mp4 format that can be played back by JavaFX. However, mp4Parser was not good for my needs, becaues it does not support "on-the-fly" conversion. It will take a lot of work to redesign mp4Parser's code for my needs. So that was my project and my code was designed to convert the file from 3rd party server and server it to JavaFX at the same time. If you need to serve a file from memory, then take a look at the following class: http://docs.oracle.com/javase/7/docs/jre/api/net/httpserver/spec/com/sun/net/httpserver/HttpServer.html
05-05-2014
Hey Roman, In order to multiplex my raw H264 and audio (PCM-U) I use MP4Parser (https://code.google.com/p/mp4parser/) which, as long as you use a ByteBuffer is REALLY fast (make sure you allocate the exact size upfront). Here's the code (note that audio is commented out because it doesn't support PCM-U and I haven't spent time to convert it yet)... H264TrackImpl h264Track = new H264TrackImpl(new MemoryDataSourceImpl(videoData)); //AACTrackImpl aacTrack = new AACTrackImpl(new MemoryDataSourceImpl(audioData.toByteArray())); Movie m = new Movie(); m.addTrack(h264Track); //m.addTrack(aacTrack); Container container = new DefaultMp4Builder().build(m); ByteOutputStream movieData = new ByteOutputStream(); WritableByteChannel writableByteChannel = Channels.newChannel(movieData); container.writeContainer(writableByteChannel); return movieData.getBytes(); ...I am currently trying to get my video hosted through a HTTP server or a simple socket, if possible. If you could please share your code that achieved this, that would be great. I am actually trying to get multiple videos (up to 4) to play at the same time (synchronized) using Media and I am seeing a memory problem (3x 1 minute videos is upwards of 300 MB already). So not only do I need to load my video from memory, but I also need to only have a part of it loaded at any given time. I was pretty close to hacking a solution together without using a server, but could not get a custom ContentHandler injected. See my SO here (http://stackoverflow.com/questions/23299486/java-net-urlconnection-setcontenthandlerfactory-doesnt-work), thanks.
25-04-2014
If there is a bug we'll fix it. Generally HLS is supported. Probably you create HLS stream server incorrectly. HLS uses MPEG TS as the transport stream.
18-10-2013
Hm... HTTP Live streaming does not seem to work in 2.2.40, though. See RT-33534. What HLS server do you test it with?
18-10-2013
Playback is supported through HTTP Live streaming. Follow this manual of creating HLS stream that's playable by the Media component: https://developer.apple.com/library/ios/documentation/networkinginternet/conceptual/streamingmediaguide/DeployingHTTPLiveStreaming/DeployingHTTPLiveStreaming.html#//apple_ref/doc/uid/TP40008332-CH2-SW3
18-10-2013
I've given it some further thought and testing. To my extreme pleasure JavaFX does support fragmented mp4. But what is the story with MPEG_TS? Is it already supported in the latest release (2.2.40) or are you just planning to support it?
18-10-2013
Flash uses f4v (that is basically mp4) for live streaming just fine, afaik. There is also a thing called "fragmented mp4". See this articles, for example: http://blogs.iis.net/samzhang/archive/2011/12/02/fragmented-mp4-vs-mpeg2-ts.aspx
17-10-2013
This is exactly because we have all file available and can build the full index for it and record it in the MP4 container. If we're talking about live streaming then MP4 will not work.
17-10-2013
My 5 cents. The proposed API is going to simplify the lives of JavaFX users. It is not a huge improvement, but it is an improvement anyway. Without this API I have to write maybe 2 pages of code for a simple http server that I can point JavaFX media to (or use an embeddable http server like Jetty if I do not care about size of the libraries). The bigger problem is explained RT-33628. When I have raw h246/vp6/mp4/aac frames, then I am forced to multiplex them into iso media container format (mp4) in order to pass them to JavaFX. That is _a_real_pain_. Due to sheer complexity of the format this is not an easy task to do. I challenge anyone to solve it in a few pages of code. I would be glad to hear if that is possible (I can contribute my two-page http server in return). There was an assumption up there in discussion: {quote} MP4 playback requires: 1) Recording to be finished (correct header with index table saved in the file). In this case we can play it from a file just like new File("somefile.mp4).getInputStream() with length provided: MPEG4 streamType. 2) Using MpegTS protocol. In this case we can play live sources and the length can be -1 or unknown to signal a live source: MPEG_TS streamType. {quote} That is not correct. MP4 playback does not require recording to be finished. MP4 stream can be optimized for progressive download and have all its indices at the beginning. Just open JavaFX Media classic "Big Bucks Bunny" mp4 from here, for example: http://download.blender.org/peach/bigbuckbunny_movies/BigBuckBunny_320x180.mp4 and observe that playback starts almost immediately while file still loads. MP4 can be used for live stream, too. You don't have to use MpegTS as the comment above implies.
17-10-2013
We added new API setEnableProgressBuffer(). By default it will be set to true, since it is required for seeking and stopping playback if seek functionality is not implemented in MediaInputStream. When progress buffer is enabled media data will be cached in file on local hard drive. If user does not want to cache media data, then it can be set to false, but then seek should be implemented, otherwise we will reject MediaInputStream with progress buffer set to false and seek not implemented. Added int read(byte[] data, long position) for advanced functionality such as metadata extraction. We require additional API for such functionality, because metadata parsing is done by Media object separately from playback. Any comments?
05-10-2013
MediaInputStream_v3.java - new version of MediaInputStream which is fully implemented. MediaInputStreamHelper.java - shows how to use MediaInputStream.
05-10-2013
We can also use mime types. We can define some set that we support as String. Use null for auto detect. This will give ability to user to specify any mime types.
14-09-2013
We can add StreamType.AUTO_DETECT, if user does not know what stream he or she tries to play. Then we can try to detect it automatically. Or it can be useful if user keep switching between like MP3 and MP4 on-demand streams, then it will not be issue for us to detect on-demend MP3 or MP4. I still believe we need a way to explicitly tell us what stream is. AUTO_DETECT can also be used for user installed codecs or CUSTOM_STREAM, then we can load codecs that user registered with us.
13-09-2013
I do not have permission to view RT-18402.
13-09-2013
We parse file header and/or extension to detect type. Live sources may not contain any headers. Headers exist of you supply input stream from a file and if seeking is available. GStreamer has typefinder element which tries to detect stream type but even this fails when we deal with live sources. For live sources GStreamer needs explicit stream type declaration. If it fails to parse it then - it ends with a error but it does not detect stream type automatically. Conclusion: If you supply a file via input stream we can do our best to detect stream type (it will depend on RT-18402) but it won't work for live sources.
13-09-2013
Agreed with Dr. Michael Paus. Besides, I don't understand at all, why have we to specify media type explicitly? It would be more fine if `Media` detects media type automatically, like it does it currently with files: new Media("file:///C:/video.mp4"); // No explicit media type new Media(new MediaInputStream(new FileInputStream("C:/video.mp4"), length, StreamType.MP4)); // Explicit media type, why???
13-09-2013
Is it a good idea to have the stream type encoded as an enum? What would you do if JavaFX should ever support user installed system codecs (as the old JavaFX 1.x did)? There should also be stream types for all currently supported media types and not just mp3/4.
13-09-2013
Attached is second version of MediaInputStream: - Added MP4. - Added default constructor with InputStream as well as implementation (not perfect) for read, seek and close. - Seek is implemented as reset() and skip() not best solution, but should work as default.
13-09-2013
Kirill, I don't agree that current demand is for MP3 only. Our company needs some way to "transparently" decrypt encrypted MP4/AVC video file. And MediaInputStream will be excellent solution for such problem. Currently we have to organize HTTP streaming on localhost to do something like this. It is *very* unconvenient and complex solution with some inherited limitations. So we need MediaInputStream not only for MP3, but also for video streams. Thanks!
12-09-2013
How about having a simple MediaInputStream(InputStream is, long length, StreamType streamType) constructor in the first version ? We can add MPEG_TS as streamType later and users will implement their own transport systems. Also seek method can be helpful if we need seek capability. We can create a default implementation of MediaInputStream with default(empty) implementation of seek(), such sources will be non seekable. Others can extend it. Media class will have one extra constructor: Media(MediaInputStream). We don't want to add many constructors with similar meaning to it. We want to concentrate various configurations of streams in one class that's independent of Media. Later we can add audioStream and videoStream which would supply the API with timestamped video (H264) and audio (AAC) streams directly from microphone and camera. Current demand is for MP3 only, so let's start with simple things. Opinions ?
12-09-2013
Comments on MediaInputStream: - Will mp3 really be the only StreamType? - Will there be a default implementation for the simple InputStream case without direct access or will the Media class provide a corresponding constructor? - What is the getFrame() method mentioned in the comment for close()?
12-09-2013
MP4 playback requires: 1) Recording to be finished (correct header with index table saved in the file). In this case we can play it from a file just like new File("somefile.mp4).getInputStream() with length provided: MPEG4 streamType. 2) Using MpegTS protocol. In this case we can play live sources and the length can be -1 or unknown to signal a live source: MPEG_TS streamType. For limited sources that have not -1 length seeking can be possible. But in the default implementation seek will be empty because seek implementation depends on your own protocols. We will update MediaInputStream little later.
12-09-2013
Attached is proposed draft of MediaInputStream that will be used to provide input to Media object. Following constructor for Media will be used: public Media(MediaInputStream mediaStream)
12-09-2013
Random access speeds up seeking. Based on the knowledge that the source is randomly accessible we use different approach for seeking. If it's not important let's skip it. Duration can be set to the real duration if this information is available in the stream or to be Duration.UNKNOWN. Good question about the source. It can be an Object and return String or InputStream.
04-09-2013
Same opinion as Dr. Paus. I don't even know, what random access is useful for. So keep things simple in the first step. What I am hoping to do with an InputStream constructor is to be able to pass an audio stream to the media object and then be able to mute that stream or change volume with the MediaPlayer and eventually be able to do a voice chat. If we need th ConnectionHolder for that, then I prefer this variant ;-). I only wonder, what happens to properties like getSource() or getDuration(). Would it return null and Duration.INDEFINITE when the media is based on a stream?
04-09-2013
Well it's even easier for us. And for SQE. Let's see if there are other opinions.
04-09-2013
In order to keep simple things simple, I'd suggest to implement at least the InputStream constructor. In addition it would of course also be nice to have the more flexible ConnectionHolder constructor but people should not be forced to implement that if all they need is just the simple InputStream variant.
04-09-2013
Having a simple InputStream is not as convenient as it may seem to be. Any kind of random access is not supported by a simple InputStream. In some situations random access allows to speedup seek performance. There is a class in our internal APIs called ConnectionHolder which gives different ways to access arbitrary content. See attached ConnectionHolder.java for reference. Users will implement the interface and we will add a Media(ConnectionHolder holder) constructor. If it's too much of a burden to everyone then we will add a simple Media(InputStream) constructor. Any comments are welcome.
04-09-2013