[Gstreamer-openmax] Discussion on the hardware accelerator solution in GstOpenMAX project.

[Ling S. <sh...@gm...>]

Hi, all
I'm in a research project to port gstreamer into embedded system. Now, we
encounter the issue on how to integrate hardware accelerators (DSP/GPU) into
gst. After evaluating different solution, we think GstOpenMAX project may be
the best one for us, because

  1) OpenMAX is an industry standard
  2) more and more DSP/GPU vendor support OpenMAX

But, I still have several questions on this project.
1. Does Nokia N8xx serials use GstOpenMAX?
I know Nokia engineers lead this project. I also know N8xx serials use
gstreamer as default playback engine, and it uses TI OMAP 2420, which has
DSP. Can anyone tell me if N8xx use GstOpenMAX? If no, does N8xx plan to use
it in the future?

2. What's GstOpenMAX plan to support DSP/GPU in the future?
I review several plugins in GstOpenMAX and find current design can only
support none-tunnel communication.  It's not the best solution in hardware,
because of bad performance. So, we plan to improve it by adding tunneled or
proprietary communication. Do you have such plan? If yes, can we involve in
design?

In addition, most accelerators work as a decoder and a render. It means, the
encoded data sent to it will directly be decoded and rendered, and will not
be retrieved back again. How the gst or omx organize its pipeline in this
situation? We are evaluating two solutions.

===Solution 1===
We can design a super omx sink component to cover decoder and render. This
is the solution is used by N8xx.
src ! demux ! sink
                         |
                       super omx sink
                         |
                   +--------------------------------------------+
                    |    hardware accelerator                 |
                   +--------------------------------------------+

===Solution 2===
We can separate omx decoder, omx post processer, and omx sink elements. We
enhance decoder, post processor, and sink plugin in GstOpenMAX. If
GstOpenMAX plugin found its neighborhood are GstOpenMAX plugin, it will try
to establish tunneled communication or proprietary communication firstly. It
means, although we have 3 OMX plugin in gst pipeline, there is no data in
gst pad and omx port. The last two gst/omx plugin only provide control
function, but not support process data flow. Of cause, If the connection is
failed, it will use none-tunnel communication.

src ! demux ! decoder ! post processor ! sink
                            |                     |                  |
                     omx dec          omx pp       omx sink
                            |                     |                  |
                   +--------------------------------------------+
                    |    hardware accelerator                 |
                   +--------------------------------------------+

It seems solution 2 is more flexible. How about your suggestion on the 2
solutions? Which one is feasible? Do you have other solutions?

Thank you very much.