Audio tracks and frame format
Brought to you by:
kspes
Menu
▾
▴
Hi. I have a couple of questions about APIs that I was unable to locate:
information about the tracks (to select language, etc)? Is it possible
to retrieve a subtitle track?
stream, in case we have an alternative way to convert it, and avoid
the call to "decodeRGBA" or whatever? I noticed on the AV playback
demo that it sets the output to TH_BGRX for mp4 videos (I would have
expected YUVX to be the frame format?), to take advantage of the gl
extension.
Thanks!
Ariel.
1) Eh, not really, I never worked on files with multiple tracks though. Same for subtitles. Can you get me a sample file so I can look into it? It would probably not be a big deal to do.
PS: Do you really need subtitles or are you just asking?
2) well, as far as theora goes, it's almost always YUV420. you can use TH_YUV which is an interleaved (not planar) YUV 4:4:4 format. I can change the code to allow users to intercept the frame decoding process and suplement it with their own. What are you trying to achieve btw? just a different RGB conversion or something specific?
if it's just RGB, you can freely use TH_RGB if you compile the library with libyuv support which is by default, it's very fast. almost as fast as copying TH_YUV data to the queued frame.
(sorry, I didn't see this reply)
1) I'm not just asking :-) I have a video with multiple audio tracks and subtitles, I need to get the language of each track and match it with my game's configuration, and use that. I saw on TheoraVideClip_AVFoundation.mm that it actually picks track 0 for audio, so the code is there, just missing some public API. Example video here:
https://godot.blob.core.windows.net/builds/depo_intro.mp4
2) in this case I'm just asking :) The hardware might support a format that matches the format of the frames, or there we might have a shader that can do the conversion on the GPU, stuff like that.
1) Ok, I'll see what I can do! thanks for the example video.
2) Yeah, actually the demos use an example YUV->RGB conversion via opengl shaders as well. In this case you can use TH_YUV format. But as I've said, in the past year, I've made significant CPU optimizations in this format conversion process so if you're able to use these cpu features on your platform, you probably don't need shaders for YUV conversion.
@Areil: oh, I see you're from Daedalic! My favorite adventure game studio besides my own :) I'll this high on my priority then :D
Haha thanks. But actually I'm not, I'm from Okam Studio (our engine is http://www.godotengine.org). But they did hire us to do some work for them, hence why I'm using that video :-)
Hi Ariel,
just wanted to let you know I haven't forgotten about this. I just haven't
had the time to devote to this problem. I fully understand your requirement
and plan to integrate it in the lib, it's a good feature, both subtitles
and audio track selection. I doubt I'll have time in the next 2 months due
to deadlines at work, but I'll try. I want to do the feature properly and
systematically, not to hack it.
Krešo
On Wed, Nov 5, 2014 at 8:58 PM, Ariel Manzur amanzur@users.sf.net wrote:
Hi.. Did you ever get around to this? We ended up using the native iOS player for that particular project (https://www.youtube.com/watch?v=BIgkoEL-D_A , comes out next week I think :), but I'd still love to have this as an option (even for that same project on other platforms..).
Also in case you're curious, here is the implementation for our engine https://github.com/okamstudio/godot/tree/master/drivers/theoraplayer (a lot of people in the community are asking for a good video player, but we're missing a few small things to be able to turn it on by default.. also I haven't tried the other stuff, like sprite animations using videos, etc)
Haven't gotten arround to it yet, but I figure if you need it that badly, it shouldn't be too much work. can you send me a sample theora video with integrated subtitles? preferably wih multiple languages embedded, not sure how to encode those in theora.
Curious, what features are you missing? I'm always glad to spread the lib to as much users as possible :)
Hi.. Video link in PM :)
Right now the main things we're missing is proper configuration of the build system to build on all platforms (I think none of them can build FFmpeg, but most of them can build the rest), plus the right compiler flags to get the right optimizations, especailly for mobile. We try not to have external references unless we have to (platform SDKs, etc), so we include the libraries we use, and we also have to share ogg and vorbis, so we have to build with our build system..
Also the a/v sync is not perfect.. And I think I hacked a temporary solution to the audio track problem, and a couple of things related to memory allocation (we use our own allocators instead of new/delete).
Hi Ariel,
Thanks! MP4 will be useful for this feature as well.
Do you really need ffmpeg? We've been able to optimize theora on most systems very well, it should be more than fast enough. You can use it on iOS as well, no need for MP4 anymore in my opinion. MP4 support was developed before we introduced arm assembly optimizations and libyuv support.
sure, you can easily shary any ogg/vorbis/tremor without problem. also it's not a problem to just clone the lib snapshot in your own repo and update from time to time.
In any case, I'd like to work on the issues you have with the lib to be able to include it in the default godot builds. Let's start with audio tracks and subtitles and work our way further.
progress update, I've been able to encode an ogv file with multiple audio and subtitle tracks (it wasn't easy :). Started working on ogg-skeleton support to identify all the streams and maybe optimize file duration analysis if ogg skeleton stream is present.
Thanks for the progress update, sounds great :) I usually use handbrake to convert, but it doesn't output ogv I think.. Do you know what is the format of the subtitles? Do you get the whole thing at the beginning, like one buffer with a whole srt file, or is it streamed along with the rest?
We don't really need ffmpeg, I just tried to compile it because it was there :-) But if there's good support for one format, that's enough, and theora is the best candidate since it can do all the advanced stuff like work as a sprite with alpha, etc. Thanks for the help :) Having a good video player as a standard feature would be great, people ask about it all the time, and we've been so close for a long time. We really appreciate this :)
yeah, handbrake is mp4 only if I'm correct. but give ffmpeg2theora a try, sure it's a command line tool but it's easy to use. As said, the lib is now good enough to handle theora videos on all platforms. so you don't have to maintain both ogv and mp4 files in your project.
btw, you can use the alpha channel in mp4 as well, it's not a codec feature, it's a feature of libtheoraplayer :)
as for subtitles, I've only reached the part where I successfully decoded the skeleton and subtitle streams, haven't played with them yet. but I assume I'll just get raw SRT data, and that shouldn't be too much of a problem to parse.
Sorry for the lack of updates, I've been immersed in a huge project with a very tough deadline.. Will resume this as soon as I can.