I'm trying to implement custom DataSource that is reading from memory - bytestream, which is supplied by some other service. I think that I'm facing two problems:
1. All custom DataSource classes should extend com.omnividea.media.protocol.DataSource otherwise you get ClassCastException since com.omnividea.media.parser.video.Parser always does this cast. This means that you cannot at the same time extend PullDataSource that enables reading from memory via PullSourceStream.read(...) method.
2. Even if I somehow find a way around this (please tell if you known a way), I'm are still facing problem, that native procedure avInit(...) accepts only string that tells which file to open (I think that other supported protocols besides file are http and ftp?). Are there any others protocols, which may be supplied from memory?
So AFAIK it is impossible to implement custom DataSource class, that is reading its data from memory, by using FOBS.
Please tell me I'm wrong.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
From what I understand, FOBS will only allow a file:// or URL such as http:// data source, you can't implement your own data source. FOBS was made to work with JMF, but it wasn't really designed from the ground up as individual JMF components, such as true JMF datasources, demuxs, codecs.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I am trying something similar. Here, an RTP streamer is running somewhere on the network. My custom datasource is busy getting data from the RTP port (UDP) and stripping out the RTP header leaving the raw data stream (it's MPEG, if you are interested).
Now, I've decided to use a push data source - which differs from your pull ds. There's some reference code at java.sun.com that shows how to build a custom data source. There's some multi-threading going on there too, to boot. It doesn't extend the fobs classes, so I'm not sure how to apply what your doing - at least not yet. Maybe I should try it.
So far, I can create a player object using my data source - but it fails to realize. This I think has to do with either fobs, or perhaps jmf - because I don't think the codec is getting called correctly. Another possibility is that my data source is out of what during the realize phase. That's where I am at today.
This is definitely like chineese water torture. I make small advances every day, but progress is VERY slow.
Personally, I don't see why fobs won't work from a custom datasource. That's just a guess, though.
Keep in touch if either of us finds something.
gene
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
The reason it doesn't work is that FOBS does the reading of the file in the c part of the code. not the java. if you search for 'bytebuffer datasource' in this forum my old attempts at solving this are to be found.
So there is a solution, but it involves writing a little bit of c as well as a bit of java. The java and c parts are both pretty straight forward though. I have actually shown a possible java side solution. Now we just need the c.. It's all explained..
Yep, that's exactly the problem I have : How to hook the java datasource up to a processor? Have you tried out your code?
Interestingly, I built a custom data source that reads a stream (either RTP or UDP and/or MPEG TS). It strips out all but the payload. If I try to hook up a processor to convert the MPEG DS into RAW type - it realizes correctly but does not read more than once. If looking at the plugin viewer on my player:
Raw Buffer -> Fobs decoder -> Fobs 2D Renderer
I think this is fine.
Now, if I hook the DS up to a player directly - stuff starts to happen. The plugin viewer shows the same arrangement as above. Reads occur constantly, and all the little flashing lights appear in the plugin viewer (data moving along). But, the screen goes completely black! I'm diggin out some bugs in the streaming portion of the code at the moment (gotta fix that first, regardless).
Thanks for the tip! I'll keep you posted on my progress.
gene
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm actually facing a similar design problem for a different Java (JNA) wrapper I am doing for ffmpeg (ffmpeg-java, a subproject of fmj, at fmj.sourceforge.net).
What I was considering doing was defining a new URLProtocol object with callbacks into java. The callbacks would, once in java, make similar calls on a JMF DataSource.
The URLProtocol would then be registered with register_protocol. Some arbitrary, non-conflicting protocol prefix would be used, like javacallback:
The JMF Demultiplexer would take the DataSource it was given, and put it in a global list, with an arbitrary ID like 1234
So then the url passed to jffmpeg to open the media would be something like javacallback://1234
The java code, when called back, would use the number to look up in a global list of DataSource. It would then know which DataSource to make its calls on.
This is just an idea, I have not tried to implement it yet.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
If you want to pass a byte[] to fobs, what format should the data be in? I've been looking into this, and it's complicated.
Let's start with an example. I know that JMF can handle MPEG-1 video natively. Sun has an example called VideoTransmit.java that takes an mpeg file on disk and streams it over the net via RTP. Their player app AVReceive2.java plays the file to the screen. They just so happen to strip out all the audio tracks, and just play the video.
I wrote an app to parse out the data coming back from the network stream (i.e. the output of the VideoTransmit.java). It sends out PES (packetized elementary streams) wrapped in RTP frames. Here's the RTP header info of interest:
1. RTP Packet size: 1468 bytes.
1. RTP Version: 2
2. RTP Payload Type: 32 (MPV-Video) aka mpeg
Unwrapping that, finds a PES header, with this info:
1. Stream ID : 0
2. Packet Length : various
Something doesn't make sense at the moment. The RTP packets are fixed at 1468 bytes long. The PES packet size is usually much larger than an RTP packet. Yet, not enough RTP packets go by to satisfy the first PES packet length before a new PES comes in. Furthermore, some of the RTP packets are non-PES and I don't know what they do yet.
The point here, is what data do you present to the FOBS engine? Is it complete PES packets? Or perhaps you have to further break down the PES packets into ES (elementary streams) before handing it off to the FOBS engine.
gene
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
thanks for this discussion, guys! Getting custom datasources work with Fobs is, along with encoding, the main lack of the current Fobs version. All the ideas on how to solve are welcomed.
However, you gotta take into account that the problem is not easy to solve and requires MAJOR modifications to the current engine. Seems like bypassing all JMF structures is still feasible in order to achieve this, but I wonder if sticking to the damn slow JMF way of doing things will end up being a better alternative. This way, the compatibility with all JMF functionality will be granted.
Hopefuly, I will be able to get someone to work on this in the future.
Cheers,
Jose San Pedro
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Just want to clarify the PES statement. I believe that when a PES packet is larger than a single RTP packet, the next N packets will NOT contain PES headers, since the packets contain data for the current PES. Now I'm wondering if there's errors in the stream if the a new PES header arrives before the previous PES packet got filled!
gene
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Ken: Can you give me a hand understanding how to use jffmpeg? I'm way lost on that. Fobs4jmf worked pretty much right out of the box for me. I'm developing in netbeans IDE, and it was pretty simple to setup (fobs, that is). But that jffmpeg really has me stumped.
Jose: Getting back to my question, do you know what type of data ffmpeg/fobs is looking for? How does it handle multiplexed audio/video? I think it must use a demultiplexor, and then 2 engines - one for video, the other for video. I realize that fobs won't work with this yet (an arbitrary datasource), but I'm trying to get a handle on this.
One last thing, do you build fobs in linux? If so, can it cross-compile to build the windows .dll files?
gene
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
fobs currently does not expect any data at all to come from the JMF side, all is done internally in the C side. However, I begun time ago the development of an experimental class called stand_alone_codec that basically makes Fobs use JMF DS and Demux, providing only the necessary codecs to decode audio and video streams. In this case, thus, the expected data will be independent multimedia streams.
This class used to work in previous versions of Fobs, but I haven't really updated it with the changes in newer releases of ffmpeg. It's definitely a nice place to start, but it comes with a bunch of problems, e.g. some information in the format headers is used by ffmpeg to initialize important fields of the codec structures which of course depend on the format/codec combinations (and that makes the code hard to do and maintain). This kind of problems are taken care of by ffmpeg if their whole decoding pipeline is used, so that's the reason I decided to do a JMF independent mode, which provides much better compatibility with video files.
About cross-compiling, there should be not many troubles. Indeed, I provide windows, linux and mac binaries available from the downloads page. Note that the windows .dll have been generated with MinGW and they are not compatible with the Visual Studio environment.
Cheers,
Jose San Pedro
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
There is another option to get this working, if the goal is simply to allow a custom DataSource.
FMJ and ffmpeg-java (which is not the same as jffmpeg) also implements a Demultiplexer (in progress), but instead of insisting on a custom datasource (com.omnividea.media.protocol.DataSource), it accepts a limited range of standard datasources:
public void setSource(DataSource source) throws IOException, IncompatibleSourceException
{
final String protocol = source.getLocator().getProtocol();
if (!(protocol.equals("file") || protocol.equals("http")))
throw new IncompatibleSourceException();
And then later, when it comes time to open the stream, it just gets the URL via source.getLocator().toExternalForm(), and passes it to ffmpeg's av_open_input_file.
In this way, any protocol supported by ffmpeg can be supported in JMF. However, this is a slight misuse of DataSource, since it just uses it to get the URL, rather than actually opening its streams.
This then opens the door to be able to use other, arbitrary datasources, as described below.
I have, however, started working on a way of creating an ffmpeg URLProtocol, which contains the callbacks to open, read, seek, etc, but have those callbacks be implemented in Java. I have gotten this to work outside of FMJ/JMF, just as a simple example. This can be seen in the ffmpeg-java (CVS) URLProtocolTest, which actually uses Java file/IO to deliver the data to ffmpeg.
As both fobs and ffmpeg-java are LGPL, feel free to borrow any code. The main difference is that fobs uses JNI and ffmpeg-java uses JNA, so calling back to java is done totally differently.
I expect to have an actual Demultiplexer for JMF/FMJ implemented using this technique relatively soon.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have successfully implemented a class that will function as an ffmpeg URLProtocol, with callbacks that do the open/read/seek etc on any JMF PullDataSource (including ones you can implement yourself).
in fmj, this is PullDataSourceCallbackURLProtocolHandler. It could be implemented very similarly in FOBS, apart from the actual native/java callbacks, as mentioned.
Ken
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
thanks a lot for the tip and the work! I'll definitely have a look at that code as a possible starting point to get it implemented in JMF+Fobs. Sincerely, given the current lack of support of JMF by SUN, I think it's good that a project such as FMJ exists. Keep on the good work!
Cheers,
Jose
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
ffmpeg-java is a different project from jffmpeg. ffmpeg-java is a subproject of FMJ. I do have it basically working, but I don't really have a packaged release for it, that could be simply dropped into JMF to work.
However, when it is done, it will be simply a matter of making sure the FMJ ffmpeg-java demultiplexer is registered in the registry (net.sf.fmj.ffmpeg_java.FFMPEGParser), and that the classpath includes the appropriate fmj and ffmpeg-java jars.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
How about push data source? A streaming source would be this type, most likely, and that's what I'm working on. Having said that, I suppose a pull data source could be made to work where the data source is is a byte[] in memory, and the stream simply fills it while the pull DS draws from it. Sound ok?
gene
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes, it seems like you could make it work with a push source as well. The code would all be similar, except that of course a read on a push source will return 0 and not block if there is no data, so you have to use the transfer handler. You effectively have to do the read in a background thread, woken up by the transfer handler, and block in the actual read that comes (ultimately) from ffmpeg.
I'll consider putting this kind of functionality into fmj/ffmpeg-java too
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm trying to implement custom DataSource that is reading from memory - bytestream, which is supplied by some other service. I think that I'm facing two problems:
1. All custom DataSource classes should extend com.omnividea.media.protocol.DataSource otherwise you get ClassCastException since com.omnividea.media.parser.video.Parser always does this cast. This means that you cannot at the same time extend PullDataSource that enables reading from memory via PullSourceStream.read(...) method.
2. Even if I somehow find a way around this (please tell if you known a way), I'm are still facing problem, that native procedure avInit(...) accepts only string that tells which file to open (I think that other supported protocols besides file are http and ftp?). Are there any others protocols, which may be supplied from memory?
So AFAIK it is impossible to implement custom DataSource class, that is reading its data from memory, by using FOBS.
Please tell me I'm wrong.
From what I understand, FOBS will only allow a file:// or URL such as http:// data source, you can't implement your own data source. FOBS was made to work with JMF, but it wasn't really designed from the ground up as individual JMF components, such as true JMF datasources, demuxs, codecs.
I am trying something similar. Here, an RTP streamer is running somewhere on the network. My custom datasource is busy getting data from the RTP port (UDP) and stripping out the RTP header leaving the raw data stream (it's MPEG, if you are interested).
Now, I've decided to use a push data source - which differs from your pull ds. There's some reference code at java.sun.com that shows how to build a custom data source. There's some multi-threading going on there too, to boot. It doesn't extend the fobs classes, so I'm not sure how to apply what your doing - at least not yet. Maybe I should try it.
So far, I can create a player object using my data source - but it fails to realize. This I think has to do with either fobs, or perhaps jmf - because I don't think the codec is getting called correctly. Another possibility is that my data source is out of what during the realize phase. That's where I am at today.
This is definitely like chineese water torture. I make small advances every day, but progress is VERY slow.
Personally, I don't see why fobs won't work from a custom datasource. That's just a guess, though.
Keep in touch if either of us finds something.
gene
Hi there team,
The reason it doesn't work is that FOBS does the reading of the file in the c part of the code. not the java. if you search for 'bytebuffer datasource' in this forum my old attempts at solving this are to be found.
So there is a solution, but it involves writing a little bit of c as well as a bit of java. The java and c parts are both pretty straight forward though. I have actually shown a possible java side solution. Now we just need the c.. It's all explained..
http://sourceforge.net/forum/message.php?msg_id=3967882
Let me know what you think.
paddy.
Hi Paddy,
Yep, that's exactly the problem I have : How to hook the java datasource up to a processor? Have you tried out your code?
Interestingly, I built a custom data source that reads a stream (either RTP or UDP and/or MPEG TS). It strips out all but the payload. If I try to hook up a processor to convert the MPEG DS into RAW type - it realizes correctly but does not read more than once. If looking at the plugin viewer on my player:
Raw Buffer -> Fobs decoder -> Fobs 2D Renderer
I think this is fine.
Now, if I hook the DS up to a player directly - stuff starts to happen. The plugin viewer shows the same arrangement as above. Reads occur constantly, and all the little flashing lights appear in the plugin viewer (data moving along). But, the screen goes completely black! I'm diggin out some bugs in the streaming portion of the code at the moment (gotta fix that first, regardless).
Thanks for the tip! I'll keep you posted on my progress.
gene
I'm actually facing a similar design problem for a different Java (JNA) wrapper I am doing for ffmpeg (ffmpeg-java, a subproject of fmj, at fmj.sourceforge.net).
What I was considering doing was defining a new URLProtocol object with callbacks into java. The callbacks would, once in java, make similar calls on a JMF DataSource.
The URLProtocol would then be registered with register_protocol. Some arbitrary, non-conflicting protocol prefix would be used, like javacallback:
The JMF Demultiplexer would take the DataSource it was given, and put it in a global list, with an arbitrary ID like 1234
So then the url passed to jffmpeg to open the media would be something like javacallback://1234
The java code, when called back, would use the number to look up in a global list of DataSource. It would then know which DataSource to make its calls on.
This is just an idea, I have not tried to implement it yet.
Here's some more gruel to chew on . . .
If you want to pass a byte[] to fobs, what format should the data be in? I've been looking into this, and it's complicated.
Let's start with an example. I know that JMF can handle MPEG-1 video natively. Sun has an example called VideoTransmit.java that takes an mpeg file on disk and streams it over the net via RTP. Their player app AVReceive2.java plays the file to the screen. They just so happen to strip out all the audio tracks, and just play the video.
I wrote an app to parse out the data coming back from the network stream (i.e. the output of the VideoTransmit.java). It sends out PES (packetized elementary streams) wrapped in RTP frames. Here's the RTP header info of interest:
1. RTP Packet size: 1468 bytes.
1. RTP Version: 2
2. RTP Payload Type: 32 (MPV-Video) aka mpeg
Unwrapping that, finds a PES header, with this info:
1. Stream ID : 0
2. Packet Length : various
Something doesn't make sense at the moment. The RTP packets are fixed at 1468 bytes long. The PES packet size is usually much larger than an RTP packet. Yet, not enough RTP packets go by to satisfy the first PES packet length before a new PES comes in. Furthermore, some of the RTP packets are non-PES and I don't know what they do yet.
The point here, is what data do you present to the FOBS engine? Is it complete PES packets? Or perhaps you have to further break down the PES packets into ES (elementary streams) before handing it off to the FOBS engine.
gene
btw, ffmpeg does appear to have support for urls that start with rtp:
So it may be that you can give ffmpeg the right url and it will magically work... I have never tried this though.
Hi,
thanks for this discussion, guys! Getting custom datasources work with Fobs is, along with encoding, the main lack of the current Fobs version. All the ideas on how to solve are welcomed.
However, you gotta take into account that the problem is not easy to solve and requires MAJOR modifications to the current engine. Seems like bypassing all JMF structures is still feasible in order to achieve this, but I wonder if sticking to the damn slow JMF way of doing things will end up being a better alternative. This way, the compatibility with all JMF functionality will be granted.
Hopefuly, I will be able to get someone to work on this in the future.
Cheers,
Jose San Pedro
Just want to clarify the PES statement. I believe that when a PES packet is larger than a single RTP packet, the next N packets will NOT contain PES headers, since the packets contain data for the current PES. Now I'm wondering if there's errors in the stream if the a new PES header arrives before the previous PES packet got filled!
gene
Ken: Can you give me a hand understanding how to use jffmpeg? I'm way lost on that. Fobs4jmf worked pretty much right out of the box for me. I'm developing in netbeans IDE, and it was pretty simple to setup (fobs, that is). But that jffmpeg really has me stumped.
Jose: Getting back to my question, do you know what type of data ffmpeg/fobs is looking for? How does it handle multiplexed audio/video? I think it must use a demultiplexor, and then 2 engines - one for video, the other for video. I realize that fobs won't work with this yet (an arbitrary datasource), but I'm trying to get a handle on this.
One last thing, do you build fobs in linux? If so, can it cross-compile to build the windows .dll files?
gene
Hi,
fobs currently does not expect any data at all to come from the JMF side, all is done internally in the C side. However, I begun time ago the development of an experimental class called stand_alone_codec that basically makes Fobs use JMF DS and Demux, providing only the necessary codecs to decode audio and video streams. In this case, thus, the expected data will be independent multimedia streams.
This class used to work in previous versions of Fobs, but I haven't really updated it with the changes in newer releases of ffmpeg. It's definitely a nice place to start, but it comes with a bunch of problems, e.g. some information in the format headers is used by ffmpeg to initialize important fields of the codec structures which of course depend on the format/codec combinations (and that makes the code hard to do and maintain). This kind of problems are taken care of by ffmpeg if their whole decoding pipeline is used, so that's the reason I decided to do a JMF independent mode, which provides much better compatibility with video files.
About cross-compiling, there should be not many troubles. Indeed, I provide windows, linux and mac binaries available from the downloads page. Note that the windows .dll have been generated with MinGW and they are not compatible with the Visual Studio environment.
Cheers,
Jose San Pedro
There is another option to get this working, if the goal is simply to allow a custom DataSource.
FMJ and ffmpeg-java (which is not the same as jffmpeg) also implements a Demultiplexer (in progress), but instead of insisting on a custom datasource (com.omnividea.media.protocol.DataSource), it accepts a limited range of standard datasources:
public void setSource(DataSource source) throws IOException, IncompatibleSourceException
{
final String protocol = source.getLocator().getProtocol();
if (!(protocol.equals("file") || protocol.equals("http")))
throw new IncompatibleSourceException();
And then later, when it comes time to open the stream, it just gets the URL via source.getLocator().toExternalForm(), and passes it to ffmpeg's av_open_input_file.
In this way, any protocol supported by ffmpeg can be supported in JMF. However, this is a slight misuse of DataSource, since it just uses it to get the URL, rather than actually opening its streams.
This then opens the door to be able to use other, arbitrary datasources, as described below.
I have, however, started working on a way of creating an ffmpeg URLProtocol, which contains the callbacks to open, read, seek, etc, but have those callbacks be implemented in Java. I have gotten this to work outside of FMJ/JMF, just as a simple example. This can be seen in the ffmpeg-java (CVS) URLProtocolTest, which actually uses Java file/IO to deliver the data to ffmpeg.
As both fobs and ffmpeg-java are LGPL, feel free to borrow any code. The main difference is that fobs uses JNI and ffmpeg-java uses JNA, so calling back to java is done totally differently.
I expect to have an actual Demultiplexer for JMF/FMJ implemented using this technique relatively soon.
I have successfully implemented a class that will function as an ffmpeg URLProtocol, with callbacks that do the open/read/seek etc on any JMF PullDataSource (including ones you can implement yourself).
in fmj, this is PullDataSourceCallbackURLProtocolHandler. It could be implemented very similarly in FOBS, apart from the actual native/java callbacks, as mentioned.
Ken
Ken,
thanks a lot for the tip and the work! I'll definitely have a look at that code as a possible starting point to get it implemented in JMF+Fobs. Sincerely, given the current lack of support of JMF by SUN, I think it's good that a project such as FMJ exists. Keep on the good work!
Cheers,
Jose
Jose -
by the way, did you ever have a chance to look at my patch for fobs that allows it to be compiled under 64-bit linux?
Thanks,
Ken
Ken,
I had forgotten about this!! Thanks for the patch and for remembering me. I'll include these changes this evening (hopefully)!
Cheers,
Jose
ffmpeg-java is a different project from jffmpeg. ffmpeg-java is a subproject of FMJ. I do have it basically working, but I don't really have a packaged release for it, that could be simply dropped into JMF to work.
However, when it is done, it will be simply a matter of making sure the FMJ ffmpeg-java demultiplexer is registered in the registry (net.sf.fmj.ffmpeg_java.FFMPEGParser), and that the classpath includes the appropriate fmj and ffmpeg-java jars.
Ken,
How about push data source? A streaming source would be this type, most likely, and that's what I'm working on. Having said that, I suppose a pull data source could be made to work where the data source is is a byte[] in memory, and the stream simply fills it while the pull DS draws from it. Sound ok?
gene
Yes, it seems like you could make it work with a push source as well. The code would all be similar, except that of course a read on a push source will return 0 and not block if there is no data, so you have to use the transfer handler. You effectively have to do the read in a background thread, woken up by the transfer handler, and block in the actual read that comes (ultimately) from ffmpeg.
I'll consider putting this kind of functionality into fmj/ffmpeg-java too