[xine-devel] The Final Frontier (Part I)

[Michael R. <mr...@us...>]

Hi team,

I am presenting the first results of my work towards a post effect 
plugin layer. I have attached several patches against current cvs for 
public review (I broke it up into multiple patches for readability). 
They may look quite heavy but they really are not. Most of it is just 
type renaming, additional parameters to some internal calls and version 
number increasing.

But I will start from the beginning and explain what I did:
The overall goal is a powerful and flexible layer of post effect plugins 
(I will skip the "effect" from here and call them "post plugins"). I 
intend the post plugin layer to be added after the decoder layer, so 
that any frame or audio buffer, that comes out of a decoder goes 
through an arbitrary tree of post plugins before it reaches the output 
layer. This way, it will be possible to do anything with the frames and 
audio buffers we want. Additionally it should be possible for a post 
plugin to have not only one input and one output, but any number of 
inputs (be it audio or video) and any number of outputs.

<cut and paste from another mail by me>
As the engine currently works, there is only a 1:1 mapping between 
streams and outputs possible. You can have multiple streams at once, 
but each with its own output. This is current status graphically:

           stream part         |      output part
stream1:
input->demuxer------->decoders--->video/audio out + overlay manager
stream2:
input->demuxer------->decoders--->video/audio out + overlay manager
stream3:
input->demuxer------->decoders--->video/audio out + overlay manager
                               ^
For certain applications, it is interesting to allow different 
stream:output mappings apart from 1:1. And this is where the post 
plugins appear: We cut through the engine at the position (^) I marked 
above. Then we create a new plugin layer that can interconnect any 
number of inputs (streams) with any number of outputs.
So the goal is this:

           stream part         |      output part
stream1:
input->demuxer------->decoders->->video/audio out + overlay manager
stream2:
input->demuxer------->decoders-+->video/audio out + overlay manager
stream3:                       |
input->demuxer------->decoders-/
</cut and paste>

Magnification of the relevant part here:
                        +------------------------+
   decoder1 --------->  | input1                 |
                        |                output  | ----> output plugins
   decoder2 --------->  | input2                 |
                        +------------------------+
                              post plugin

With a design that allows any combination of input and output, a number 
of interesting applications can be achieved:
* visualization plugins
  (1 audio input, 1 audio and 1 video output; audio is simply passed on,
  video is generated from the audio data)
* separate subtitles in separate streams
  (2 video inputs, 1 video output; video from the first and overlays
  from the second input are passed on)
* audio spectrometer
  (1 audio input, 1 video output)
* vectorscope visualisations of video colorspace
  (1 video input, 1 video output)
* alphablending
  (3 video inputs, 1 video output; inputs 1 and 2 are blended with
  input 3 as an alpha channel)
...

Here comes the way we accomplish all that:
Part I: Modify the engine, so that stream and output parts are 
independent as illustrated above. One output should be able to handle 
various streams.
Part II: Design an API for the post plugins.
Part III: Write a first, simple post plugin to see, if things work.

My patches realise Part I and I already have Part II and III worked out
in my mind, so I hereby volunteer to go all the way through and present 
some solutions on the list.

This is what the patches do in detail:
* public_API.patch
Sorry, but I have to modifiy the public API a bit. But don't worry, 
source and binary compatibility is maintained. The idea is to abstract 
from xine_{vo,ao}_driver_t and introduce xine_{audio,video}_port_t. 
Such a port would be provided by a driver and used by a stream as with 
the former xine_{vo,ao}_driver_t structure. But in addition to that, a 
post plugin would take a number of ports on init (which it connects to 
its output) and it provides a number of ports in return (its inputs). 
So this is kind of a plug&socket concept. The old structures 
xine_{vo,ao}_driver_t are now convenience aliases to 
xine_{video,audio}_port_t.

* engine.patch
This is the heart of the patch. It implements three major modifications: 
Provide the xine_{audio,video}_port_t structures, make outputs handle 
multiple streams and introduce a global clock.

The xine_{audio,video}_port_t structures are the former internal 
{ao,vo}_instance_t structures, since they provide the methods that link 
streams and outputs. So the port is basically just an abstraction layer 
(like its ancestor {ao,vo}_instance_t) providing calls to put data on 
the out fifos. The actual output plugin is then used by the video/audio 
out loop to do the actual work:

                    +-------------+       +---------------+
  decoder --------> | output loop | ----> | output plugin |
                    +-------------+       +---------------+
                    ^                     ^
                position of new         position of old
         xine_{audio,video}_port_t     xine_{ao,vo}_driver_t
                interface               interface

So everything is as before, we just hand out another interface to the 
public (and to the streams). The whole idea behind this is the 
following:

              +-------------+     +-------------+      +---------------+
  decoder --> | post plugin | --> | output loop | ---> | output plugin |
              +-------------+     +-------------+      +---------------+
              ^                   ^
            these two interfaces are both the
            same xine_{audio,video}_port_t interface

This way it will be transparent to the decoders whether they output to a 
post plugin or directly to the output.
So at first the engine patch renames {ao,vo}_instance_t to 
xine_{audio,video}_port_t. xine_{ao,vo}_driver_t is renamed back to 
{ao,vo}_driver_t, since this is no longer public.

Additionally we want the output to handle multiple streams, so direct 
calls from the output loops into the stream are changed: Every decoder 
registers its stream to its output with the call to 
{audio,video}_out->open(). This call is already done by the decoders, 
it just passes the stream as an additional parameter. This way, the 
output loops can maintain a list of all currently assigned streams and 
then any call to the streams (like decoder flushing in video out loop) 
is iterated over all streams. Vice versa {audio,video}_out->close() 
unregisters the stream. On functions involving a metronom of one 
specific stream, a new stream parameter was added too. (This means 
img->draw() and audio_out->put_buffer() have additional stream 
parameters).

Now we come to metronom and thus to the third change: Since the (local) 
metronoms are stream specific, we need a global clock to get the time 
from in the output loops. Therefore I moved all SCR specific parts out 
of metronom_t, which now only holds the stream specific part, into a 
global metronom_clock_t which resides in xine_t. This way, all streams 
are automatically synced to the same clock and we can easily ask for 
current time inside the output loops. The major disadvantage of this is 
that it is no longer possible to play different streams at different 
speeds, but given the benefits, I think we can live with that for now. 
We can implement different solutions for multispeed later. And if any 
application really needs to have streams with different clock speeds, 
it can still open multiple xine_t instances with each having its own 
clock.

* decoders.patch
This one adds the additional parameters to {audio,video}_out->open(), 
{audio,video}_out->close(), img->draw() and audio_out->put_buffer().
I skip bumping up the interface version numbers here, because Miguel did 
that with his recent metronom patch.

* output.patch
This bumps up interface version numbers and renames 
xine_{ao,vo}_driver_t to {ao,vo}_driver_t.

* input.patch, demuxers.patch
These only bump up the interface version numbers.

I hope I made everything a bit clearer and I hope we can agree on all 
this. I would like to commit it to cvs ASAP (so that I don't have to 
spend so much time resolving cvs conflicts ;). The question is just: Do 
we want to make a new release first?
Of course, if anyone has any serious objections or questions, we shall 
discuss things first.

Michael

-- 
# Basic IBM dingbats, some of which will never have a purpose clear
# to mankind
	2.4.0 linux/drivers/char/cp437.uni