Re: [Mlt-devel] OpenGL support proposal
Brought to you by:
ddennedy,
lilo_booter
From: Christophe T. <hf...@fr...> - 2012-03-30 21:45:55
|
Le 30/03/2012 21:07, Andrew Wason a écrit : > Christophe Thommeret<hftom<at> free.fr> writes: >> This is a request for review. >> Anyone interested can clone http://hftom.homelinux.org/mlt.git >> Look at src/modules/opengl/README. > > This looks interesting. I'm working on something complementary. We > have mostly been using WebVfx to implement transitions/filters with > WebGL (e.g. Instagram-style video filters, 3D model transitions with > the video textured into the model - see > http://docs.motionbox.com/videos/transition-BabyRattle.mp4 etc.) Very nice! > Using > WebGL via WebKit/WebVfx isn't that efficient (and the future of the Qt > webkit1 hybrid APIs required is in doubt), so I'm writing a WebGL > implementation directly on top of the V8 JavaScript engine (v8-webgl). > > The plan is to be able to write new MLT filters/transitions in > JavaScript using WebGL (leveraging frameworks like Three.js), then use > that *.js file as a resource for the MLT filter/transition plugin. > > It would be nice to be able to leverage what you're doing to keep the > mlt_image on the gpu as much as possible. Right now my plugin would > have to take the mlt_image, convert to rgb(a) and upload to the gpu to > render a frame, then glReadPixels it back to the mlt_image to pass > down the filter chain. > > I almost always am using the avformat consumer to render the video > into a file, so I wouldn't be relying on the consumer to init the GL > context (e.g. like the xgl consumer). So my plugin could install a > context using mlt_glsl_init if one doesn't exist, or use the existing > one if some other plugin has already installed it (e.g. the xgl > consumer plugin). But I think you would need to expose at least > lock/unlock on glsl_env. glsl_env should probably be part of mlt core > instead of the opengl plugin - so that any plugin that wants to > process mlt_image_glsl format images can bind the correct GL context. Yes, i think it would be possible to have mlt_glsl.h being a public header. Not only lock/unlock would be used but probably also gl objects allocation, since glsl_env is supposed to manage gpu resources (althought not yet completely implemented). The opengl plugin will also have code to create gl context itself, since this is required for offscreen rendering (e.g. using avformat). The application (e.g. melt) would have a cli option to do so (for example by calling mlt_glsl_init( NUL, NULL, NUL) that would mean "no shared display context, create your own for rendering"). > So what I'm hoping to be able to do is use a generic application like > melt to filter a video through multiple GPU based filters/transitions > (both your glsl filters like glsl.rescale etc., and my v8-webgl > filters), then finally the mlt_image_glsl format image would be > consumed by the avformat consumer, where it would be downloaded from > the GPU and converted (via glsl.csc) to mlt_image_yuv420p or something > and written to a file. > > I need MSAA enabled on the GL context, but if my plugin is the one > calling mlt_glsl_init then I guess I can enable it then. If some other > plugin (or the frontend) has created the context, then that might be a > problem. After rendering a frame I would need to resolve the FBO by > blitting it to a non-multisampled FBO bound to the mlt texture. So I > think what you're proposing would work, but glsl_env needs to be > exposed so other plugins can take advantage of mlt_image_glsl format > images. I've already thought about MSAA and it's sure possible. Should it be a requirement or an option is questionnable. All this is work in progress anyway. > > I built your branch, but it seems to deadlock either right away or > after rendering a single frame. > > $ melt -profile atsc_1080p_25 -consumer xgl womandancingindress-16x9.mp4 > > Thread 3 (Thread 0x7fffe277f700 (LWP 7858)): > #0 0x00007ffff76c7473 in poll () from /lib/x86_64-linux-gnu/libc.so.6 > #1 0x00007ffff100e4f2 in ?? () from /usr/lib/x86_64-linux-gnu/libxcb.so.1 > #2 0x00007ffff100faaf in xcb_wait_for_reply () > from /usr/lib/x86_64-linux-gnu/libxcb.so.1 > #3 0x00007ffff3072b7d in _XReply () from > /usr/lib/x86_64-linux-gnu/libX11.so.6 > #4 0x00007fffe7ff2438 in ?? () from /usr/lib/nvidia-current/libGL.so.1 > #5 0x00007fffe7ff3e8b in ?? () from /usr/lib/nvidia-current/libGL.so.1 > #6 0x00007fffe82926d6 in hidden_done_current (user_data=0x7fffe84a0820) > Looks like you are seeing the bug described in README->Internals->Note2 ------------ Christophe Thommeret |