Hi,
I just wrote a simple app to test the performance of the fobs parser and it seems quite slow compared to mplayer. For the same file, I get 20% cpu usage at 25 fps in fobs vs 8% in mplayer for decoding + rendering. I also wrote a loop to see the max fps I could get with fobs, and I couldn't decode more than 40fps withing 100% cpu usage.
My question is the following, since my c++ skills are very limited, I can't write a benchmark for fobs w/o jmf. And so I wanted to know is the overhead mostly due to fobs or jni ?
I think writing to a direct ByteBuffer rather than an array would be faster and so if the overhead of avProcess is significant I'd like to investigate that solution.
Cheers
Masato
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Clearly, as part of the process is Java, Fobs won't ever reach mplayer performance. I still think, as you do, it still can be boosted.
A profiling in MacOSX shows that the main bottlenecks are located in:
- memcpy - 17.7%
- YUV420p_to_RGBA32 - 16.9%
The later is the color space conversion needed to get RGB channels of Frames.This step is unavoidable, although the routine to do it may be improved. Anyway, it belongs to the ffmpeg project.
The memcpy are invoked from jio_vsnprintf. Not 100% sure but very likely these come from the getIntArrayElements and releaseIntArrayElement in the Process functions. I don't have a huge JNI programming background. If you think native buffers could help, I might give it a glance.
Cheers.
Jose San Pedro
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I've tried to change the code to use a nio buffer. The cpu usage went from ~20% to ~14% (still not mplayer :( But the fps I could achieve w/o the rendering went from somewhere aroud 45 fps to around 150 fps. The problem is that I don't think any of the jmf renderer supports nio buffers, so we should either implement a lot of plugins or convert to arrays. I'm afraid that this would remove the performance gain, but it would be a good way to support both nio and standard arrays. Also java2d doesn't support creating images from nio buffers. So either render w/ jni or create a custom SampleModel (I think it would be slow).
BTW I noticed that you call releaseIntArrayElements everytime in avProcess. I've never done jni before but I wondered if it's not possible to keep a reference to the array in native code and only release the array when closing the file.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
could you share your nio-enabled version of Fobs? I really want to make some tests with it in different platforms to see if it helps more than it does with Linux.
THANKS!
Jose San Pedro
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi Robert,
I see your point, but the problem is that jni only deals with direct byte buffers, and the only way to create a direct buffer is by calling ByteBuffer.allocateDirect() or create one from a jni method. If you look at wrap() it's a static method, and will create a new (non direct) ByteBuffer around that int[], so the only way to access it from jni is to call array() on it and pass that int[] to jni. Again you lose performance due to jni copying that array back and forth.
I appreciate your interest, but it seems that the only way to create a BufferedImage from a nio buffer would be to create a custom DataBuffer. I think it would be slow though since there are only methods to access single pixels at a time. I think the best way to accelerate a nio-backed DataBuffer would be to call on Sun to create and implement such a class, then when you call drawImage, it would know that the image is backed by a nio Buffer and accelerate it accordingly.
Does anyone have a better idea? I think it would be better if we didn't have to go into jni to do the drawing.
Cheers,
Masato
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
I've done some changes to decoder.cpp and Parser.java to write into a ByteBuffer and then copy into a int[]. It seems as fast or faster than copying into an int[]. There wasn't really much to be done. Can I send my changes on the cvs. I also found out that the slowest thing was the rendering. I tried with swing, awt and swing+opengl pipeline. The opengl pipeline was really slow on my pc. The problem with swing and awt on linux is that every image is copied back and forth through X11. After reading that sun uses the Shared Memory Extension for faster rendering, I thought we could write a renderer that does the same using a Canvas and jni.
I'm not too familiar with C++ so if somebody could lend me a hand at doing it I would be very grateful. There is a sample code on how to use jni for awt, and I found this http://pantransit.reptiles.org/prog/mit-shm.html
for using shared memo extension. I might start writing this week.
Cheers,
Masato
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I have been looking at doing something like rendering to a container from jni myself, but my c++ is limited. Also, I know there is a pipeline from the renderer back to the codec that would have to be kept track of and zeroed out somehow to keep JMF parsing smoothly right? This looks very interesting and my hope is to enable better performance in fobs.
Thanks,
James
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm not so sure that filling a Java container from a native method is allowed. Anyway, we are willing to hear different potential solutions because I think we will eventually find a way to boost performance a lot. Just to let you know, I'm working with the help of Robert in a Java Renderer that will use an OpenGL renderer. He has sent me in addition a standard renderer that seems to improve performance in windows machines by a 10%. I haven't commited it yet to the CVS but I'm looking forward to hearing from your experiences with this new renderer and even with improvements to it.
Thanks everyone for your contributions!!
Cheers.
Jose San Pedro
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Yes, I don't have much experience with native rendering either, but I have been looking around and I found a few examples that write to a container from jni. I was going to take a stab at implementing:
I haven't looked at the code you just released, but everything sounds like it progressing well. I'd loved to see this boost performance on the mac too!
Thanks,
James
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
Thanks Robert and Jos for your great job!! I haven't downloaded your renderer yet. I'm on windows right now. I'll do so tonight probably. Does it work on linux? I think it's the platform that would the most benefit from it, since most of the performance bloat is due to X. The only problem with a native renderer is that (to my knowledge) it can only render into java.awt.Canvas. As I told you I was working on one that would use mit-shm on linux, but I found out c++ really isn't my cup of tea, and OGL is probably faster. How hard would it be to port that renderer to linux?
Thanks again
Masato
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
We've used the jni canvas extention thing to get the X Window ID of the awt canvas, then told mplayer to render to that window id, and volia! video in a java app. Perhaps you could use
the same approach.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Do you have any of the sample c code that you use that you could post to help out? I have no problems getting the window ID or anything like that in JNI, but I get lost a little in the rendering, and converting to an image, such as how they render in ffplay. Maybe mplayer is different or easier to follow.
Thanks!
James
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
It really comes just in time because yesterday I began testing the new OpenGL Renderer. It's really a difference (about twice as fast) but after some profiling I think it can be vastly improved.
However, speed comes at a price. The Color Space Conversion is being performed by a Pixel Shader (CSC was the heaviest task of the process). This means that the buffer now contains YUV420P and not RGB or RGBA. This will make applying effects kind of trickier.
What do you guys think of this?!? We can use two operation modes: Fast for plain visualization and RGB-Slow for best compatibility and filter support.
If you think this is appropriate, then I think I could bypass the native-to-java memory copy and make it boost like Java was never there ;)
Cheers.
Jose San Pedro
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hi,
I just wrote a simple app to test the performance of the fobs parser and it seems quite slow compared to mplayer. For the same file, I get 20% cpu usage at 25 fps in fobs vs 8% in mplayer for decoding + rendering. I also wrote a loop to see the max fps I could get with fobs, and I couldn't decode more than 40fps withing 100% cpu usage.
My question is the following, since my c++ skills are very limited, I can't write a benchmark for fobs w/o jmf. And so I wanted to know is the overhead mostly due to fobs or jni ?
I think writing to a direct ByteBuffer rather than an array would be faster and so if the overhead of avProcess is significant I'd like to investigate that solution.
Cheers
Masato
Hi Masato,
Clearly, as part of the process is Java, Fobs won't ever reach mplayer performance. I still think, as you do, it still can be boosted.
A profiling in MacOSX shows that the main bottlenecks are located in:
- memcpy - 17.7%
- YUV420p_to_RGBA32 - 16.9%
The later is the color space conversion needed to get RGB channels of Frames.This step is unavoidable, although the routine to do it may be improved. Anyway, it belongs to the ffmpeg project.
The memcpy are invoked from jio_vsnprintf. Not 100% sure but very likely these come from the getIntArrayElements and releaseIntArrayElement in the Process functions. I don't have a huge JNI programming background. If you think native buffers could help, I might give it a glance.
Cheers.
Jose San Pedro
I've tried to change the code to use a nio buffer. The cpu usage went from ~20% to ~14% (still not mplayer :( But the fps I could achieve w/o the rendering went from somewhere aroud 45 fps to around 150 fps. The problem is that I don't think any of the jmf renderer supports nio buffers, so we should either implement a lot of plugins or convert to arrays. I'm afraid that this would remove the performance gain, but it would be a good way to support both nio and standard arrays. Also java2d doesn't support creating images from nio buffers. So either render w/ jni or create a custom SampleModel (I think it would be slow).
BTW I noticed that you call releaseIntArrayElements everytime in avProcess. I've never done jni before but I wondered if it's not possible to keep a reference to the array in native code and only release the array when closing the file.
I think there can be a possibility of moving java2d closer to nio, but I am not sure if it is faster or even works.
I think a possibility would be to take a BufferedImage that represents the current frame and create it in the size of the frame.
than create a IntBuffer the following way:
IntBuffer,wrap(( (DataBufferInt) bufferedImage.getRaster().getDataBuffer()).getData())
Now the BufferedImage should be combined with the IntBuffer so every change on the IntBuffer would also affect the BufferedImage.
Perhaps this could be a solution.
Regards, Robert
Hi,
could you share your nio-enabled version of Fobs? I really want to make some tests with it in different platforms to see if it helps more than it does with Linux.
THANKS!
Jose San Pedro
Hi Robert,
I see your point, but the problem is that jni only deals with direct byte buffers, and the only way to create a direct buffer is by calling ByteBuffer.allocateDirect() or create one from a jni method. If you look at wrap() it's a static method, and will create a new (non direct) ByteBuffer around that int[], so the only way to access it from jni is to call array() on it and pass that int[] to jni. Again you lose performance due to jni copying that array back and forth.
I appreciate your interest, but it seems that the only way to create a BufferedImage from a nio buffer would be to create a custom DataBuffer. I think it would be slow though since there are only methods to access single pixels at a time. I think the best way to accelerate a nio-backed DataBuffer would be to call on Sun to create and implement such a class, then when you call drawImage, it would know that the image is backed by a nio Buffer and accelerate it accordingly.
Does anyone have a better idea? I think it would be better if we didn't have to go into jni to do the drawing.
Cheers,
Masato
Hi,
I've done some changes to decoder.cpp and Parser.java to write into a ByteBuffer and then copy into a int[]. It seems as fast or faster than copying into an int[]. There wasn't really much to be done. Can I send my changes on the cvs. I also found out that the slowest thing was the rendering. I tried with swing, awt and swing+opengl pipeline. The opengl pipeline was really slow on my pc. The problem with swing and awt on linux is that every image is copied back and forth through X11. After reading that sun uses the Shared Memory Extension for faster rendering, I thought we could write a renderer that does the same using a Canvas and jni.
I'm not too familiar with C++ so if somebody could lend me a hand at doing it I would be very grateful. There is a sample code on how to use jni for awt, and I found this http://pantransit.reptiles.org/prog/mit-shm.html
for using shared memo extension. I might start writing this week.
Cheers,
Masato
I have been looking at doing something like rendering to a container from jni myself, but my c++ is limited. Also, I know there is a pipeline from the renderer back to the codec that would have to be kept track of and zeroed out somehow to keep JMF parsing smoothly right? This looks very interesting and my hope is to enable better performance in fobs.
Thanks,
James
Hi,
I'm not so sure that filling a Java container from a native method is allowed. Anyway, we are willing to hear different potential solutions because I think we will eventually find a way to boost performance a lot. Just to let you know, I'm working with the help of Robert in a Java Renderer that will use an OpenGL renderer. He has sent me in addition a standard renderer that seems to improve performance in windows machines by a 10%. I haven't commited it yet to the CVS but I'm looking forward to hearing from your experiences with this new renderer and even with improvements to it.
Thanks everyone for your contributions!!
Cheers.
Jose San Pedro
Yes, I don't have much experience with native rendering either, but I have been looking around and I found a few examples that write to a container from jni. I was going to take a stab at implementing:
http://forum.java.sun.com/thread.jspa?forumID=52&threadID=404902
http://www.codeproject.com/java/opengl.asp
reference material:
http://java.sun.com/j2se/1.4.2/docs/guide/awt/1.3/AWT_Native_Interface.html
http://www.javaworld.com/javaworld/javatips/jw-javatip86.html
I haven't looked at the code you just released, but everything sounds like it progressing well. I'd loved to see this boost performance on the mac too!
Thanks,
James
Hi,
Thanks Robert and Jos for your great job!! I haven't downloaded your renderer yet. I'm on windows right now. I'll do so tonight probably. Does it work on linux? I think it's the platform that would the most benefit from it, since most of the performance bloat is due to X. The only problem with a native renderer is that (to my knowledge) it can only render into java.awt.Canvas. As I told you I was working on one that would use mit-shm on linux, but I found out c++ really isn't my cup of tea, and OGL is probably faster. How hard would it be to port that renderer to linux?
Thanks again
Masato
We've used the jni canvas extention thing to get the X Window ID of the awt canvas, then told mplayer to render to that window id, and volia! video in a java app. Perhaps you could use
the same approach.
Do you have any of the sample c code that you use that you could post to help out? I have no problems getting the window ID or anything like that in JNI, but I get lost a little in the rendering, and converting to an image, such as how they render in ffplay. Maybe mplayer is different or easier to follow.
Thanks!
James
I found some java/jogl code that someone ported for Quicktime. Looks pretty interesting.
QTGLJNI
http://www.javagaming.org/cgi-bin/JGNetForums/YaBB.cgi?board=jogl;action=display;num=1058674409;start=105
Don't if people are still thinking about a opengl renderer.
James
Hi,
hey...nnice link! Thanks a lot!!
It really comes just in time because yesterday I began testing the new OpenGL Renderer. It's really a difference (about twice as fast) but after some profiling I think it can be vastly improved.
However, speed comes at a price. The Color Space Conversion is being performed by a Pixel Shader (CSC was the heaviest task of the process). This means that the buffer now contains YUV420P and not RGB or RGBA. This will make applying effects kind of trickier.
What do you guys think of this?!? We can use two operation modes: Fast for plain visualization and RGB-Slow for best compatibility and filter support.
If you think this is appropriate, then I think I could bypass the native-to-java memory copy and make it boost like Java was never there ;)
Cheers.
Jose San Pedro