From: Zhao Liang-E. <E3...@mo...> - 2008-03-03 06:03:13
|
Hi all, On embedded device, DSP is used widely, many DSPs have different features, for example: 1. decoder DSP is just a hardware decoder 2. decoder + sink DSP is a decoder plus sink, it can directly accept encoded data, and then decode it and render pcm data into audio device directly. 3. A/V sync DSP can do A/V sync internally or not. There may be more features i don't list. For these features, how can we implement them by gstreamer? For case 1, I think it is esay to do by gstreamer, but for each 2 and 3, it seems gstreamer has no available resources to define these application. So I think we can discuss these topics and find a good solution. BTW, I find a project in gst - "gst-openmax", it is using openmax as low level implementation, but it also has some limitation, for example, no solution for case 2, and we must have openmax implementation for special DSP firstly. I hope we can have top level solution for DSP in despite of the low level implementation, just link gstbasesink virtual class, but all interfaces or virtual functions are defined to handle all the use cases. Best Regards Zhao Liang |
From: Benoit F. <ben...@pu...> - 2008-03-03 08:49:21
|
Hi, Zhao Liang-E3423C wrote: > Hi all, > > On embedded device, DSP is used widely, many DSPs have different > features, for example: > > 1. decoder > DSP is just a hardware decoder > 2. decoder + sink > DSP is a decoder plus sink, it can directly accept encoded data, and > then decode it and render pcm data into audio device directly. creating a sink with right caps should be enough for this one... > 3. A/V sync > DSP can do A/V sync internally or not. then you can ask basesink not to synchronize flows. > > There may be more features i don't list. For these features, how can > we implement them by gstreamer? > > For case 1, I think it is esay to do by gstreamer, but for each 2 and > 3, it seems gstreamer has no available resources to define these > application. So I think we can discuss these topics and find a good > solution. > > BTW, I find a project in gst - "gst-openmax", it is using openmax as > low level implementation, but it also has some limitation, for > example, no solution for case 2, and we must have openmax > implementation for special DSP firstly. > > I hope we can have top level solution for DSP in despite of the low > level implementation, just link gstbasesink virtual class, but all > interfaces or virtual functions are defined to handle all the use cases. -- Benoit Fouet Purple Labs S.A. www.purplelabs.com |
From: Zhao Liang-E. <E3...@mo...> - 2008-03-03 09:07:58
|
Zhao Liang-E3423C wrote: > Hi all, > > On embedded device, DSP is used widely, many DSPs have different > features, for example: > > 1. decoder > DSP is just a hardware decoder > 2. decoder + sink > DSP is a decoder plus sink, it can directly accept encoded data, and > then decode it and render pcm data into audio device directly. creating a sink with right caps should be enough for this one... Zhao Liang: By my experience, it is not simple like just adding a new caps. Just a example, how to handle preroll? Generally, dsp need to do initialization when start to work, so what is the time to do this? another issue is seek, how does DSP handle seek? > 3. A/V sync > DSP can do A/V sync internally or not. then you can ask basesink not to synchronize flows. Zhao Liang: If DSP is just a video decoder and sink, how does DSP sync with audio sink? maybe current basesink considers more software sink than hardware sink. > > There may be more features i don't list. For these features, how can > we implement them by gstreamer? > > For case 1, I think it is esay to do by gstreamer, but for each 2 and > 3, it seems gstreamer has no available resources to define these > application. So I think we can discuss these topics and find a good > solution. > > BTW, I find a project in gst - "gst-openmax", it is using openmax as > low level implementation, but it also has some limitation, for > example, no solution for case 2, and we must have openmax > implementation for special DSP firstly. > > I hope we can have top level solution for DSP in despite of the low > level implementation, just link gstbasesink virtual class, but all > interfaces or virtual functions are defined to handle all the use cases. -- Benoit Fouet Purple Labs S.A. www.purplelabs.com |
From: Yann G. <yan...@pu...> - 2008-03-03 09:22:30
|
Hi Zhao, Zhao Liang-E3423C wrote: > Zhao Liang-E3423C wrote: >> >> 2. decoder + sink >> DSP is a decoder plus sink, it can directly accept encoded data, and >> then decode it and render pcm data into audio device directly. >> > > creating a sink with right caps should be enough for this one... > > Zhao Liang: By my experience, it is not simple like just adding a new > caps. > Just a example, how to handle preroll? Generally, dsp > need to do initialization when start to work, so what is the time to do > this? > you can add your sink into the pipeline but not link its sink pad immediatly when creating it. Your sink element could post a message on the pipeline bus to notify it's ready when it happens, and then you link it to allow preroll. -- Yann Gilquin Purple Labs S.A. www.purplelabs.com |
From: Benoit F. <ben...@pu...> - 2008-03-03 09:30:45
|
Hi, Zhao Liang-E3423C wrote: > Zhao Liang-E3423C wrote: > >> Hi all, >> >> On embedded device, DSP is used widely, many DSPs have different >> features, for example: >> >> 1. decoder >> DSP is just a hardware decoder >> 2. decoder + sink >> DSP is a decoder plus sink, it can directly accept encoded data, and >> then decode it and render pcm data into audio device directly. >> > > creating a sink with right caps should be enough for this one... > > Zhao Liang: By my experience, it is not simple like just adding a new > caps. > Just a example, how to handle preroll? Generally, dsp > need to do initialization when start to work, so what is the time to do > this? > another issue is seek, how does DSP handle seek? > > this is generally the demuxer/parser's job to handle seek, do you have examples where it is up to the decoder to do it ? >> 3. A/V sync >> DSP can do A/V sync internally or not. >> > > then you can ask basesink not to synchronize flows. > > Zhao Liang: If DSP is just a video decoder and sink, how does DSP sync > with audio sink? maybe current basesink considers more software sink > than hardware sink. > > I was talking about your hardware handling A/V synchronization, not the opposite. buffers that are handled by sink are timestamped (by demuxers for instance) and are rightly handled by basesink when synchronization has to be done in framework, and not in HW. -- Benoit Fouet Purple Labs S.A. www.purplelabs.com |
From: Zhao Liang-E. <E3...@mo...> - 2008-03-04 02:05:31
|
Hi, Zhao Liang-E3423C wrote: > Zhao Liang-E3423C wrote: > >> Hi all, >> >> On embedded device, DSP is used widely, many DSPs have different >> features, for example: >> >> 1. decoder >> DSP is just a hardware decoder >> 2. decoder + sink >> DSP is a decoder plus sink, it can directly accept encoded data, and >> then decode it and render pcm data into audio device directly. >> > > creating a sink with right caps should be enough for this one... > > Zhao Liang: By my experience, it is not simple like just adding a new > caps. > Just a example, how to handle preroll? Generally, dsp need to do > initialization when start to work, so what is the time to do this? > another issue is seek, how does DSP handle seek? > > >this is generally the demuxer/parser's job to handle seek, do you have examples where it is up to the decoder to do it ? As it is a sink for DSP although it includes decoder functions, sink would handle preroll, in current preroll mechanism, it only allows sink do once preroll virtual function, all other buffers are pushed into queue, but some hardwares need more buffers to do initialization. Another issue is if the DSP is audio decoder + sink, how to handle clock line increasing? Currently, we can use gstaudiosink to do it, gstringbuffer can handle all clock line increasing, but some DSP can not do it if no data is fed into, so the clock calculation is really a issue. If no mechanism, rtsp streaming application will not handle buffer delay or lost. >> 3. A/V sync >> DSP can do A/V sync internally or not. >> > > then you can ask basesink not to synchronize flows. > > Zhao Liang: If DSP is just a video decoder and sink, how does DSP sync > with audio sink? maybe current basesink considers more software sink > than hardware sink. > > >I was talking about your hardware handling A/V synchronization, not the opposite. >buffers that are handled by sink are timestamped (by demuxers for >instance) and are rightly handled by basesink when synchronization has to be done in framework, and not in HW. I understood your meaning, A/V sync is really done by basesink, but some DSPs also have internal buffers to store buffers synced. and these synced buffers are encoded buffers, not raw buffers. There are 4 combinations: 1. DSP videosink v.s software audiosink 2. DSP videosink v.s DSP audiosink 3. software videosink v.s DSP audiosink 4. software videosink v.s software audiosink How to sync a/v with encoded buffers? It really puzzled me. openmax can do it well for some cases, as it has a clock in itself. but by my understanding, basesink is used to sync decoded buffers. Appreicate for your clarifying. -- Benoit Fouet Purple Labs S.A. www.purplelabs.com |
From: Zhang Yanlong-P. <PB...@mo...> - 2008-03-04 02:10:18
|
Yes, I think it is not a simple things if integrate gstreamer with other hardware, e.g dsp. Maybe implement a set of API in dsp side and provide it to gstreamer is a good solution. BRs Zhang -----Original Message----- From: gst...@li... [mailto:gst...@li...] On Behalf Of Zhao Liang-E3423C Sent: Tuesday, March 04, 2008 10:05 AM To: Benoit Fouet Cc: gst...@li... Subject: Re: [gst-embedded] How to use DSP in gstreamer Hi, Zhao Liang-E3423C wrote: > Zhao Liang-E3423C wrote: > >> Hi all, >> >> On embedded device, DSP is used widely, many DSPs have different >> features, for example: >> >> 1. decoder >> DSP is just a hardware decoder >> 2. decoder + sink >> DSP is a decoder plus sink, it can directly accept encoded data, and >> then decode it and render pcm data into audio device directly. >> > > creating a sink with right caps should be enough for this one... > > Zhao Liang: By my experience, it is not simple like just adding a new > caps. > Just a example, how to handle preroll? Generally, dsp need to do > initialization when start to work, so what is the time to do this? > another issue is seek, how does DSP handle seek? > > >this is generally the demuxer/parser's job to handle seek, do you have examples where it is up to the decoder to do it ? As it is a sink for DSP although it includes decoder functions, sink would handle preroll, in current preroll mechanism, it only allows sink do once preroll virtual function, all other buffers are pushed into queue, but some hardwares need more buffers to do initialization. Another issue is if the DSP is audio decoder + sink, how to handle clock line increasing? Currently, we can use gstaudiosink to do it, gstringbuffer can handle all clock line increasing, but some DSP can not do it if no data is fed into, so the clock calculation is really a issue. If no mechanism, rtsp streaming application will not handle buffer delay or lost. >> 3. A/V sync >> DSP can do A/V sync internally or not. >> > > then you can ask basesink not to synchronize flows. > > Zhao Liang: If DSP is just a video decoder and sink, how does DSP sync > with audio sink? maybe current basesink considers more software sink > than hardware sink. > > >I was talking about your hardware handling A/V synchronization, not the opposite. >buffers that are handled by sink are timestamped (by demuxers for >instance) and are rightly handled by basesink when synchronization has to be done in framework, and not in HW. I understood your meaning, A/V sync is really done by basesink, but some DSPs also have internal buffers to store buffers synced. and these synced buffers are encoded buffers, not raw buffers. There are 4 combinations: 1. DSP videosink v.s software audiosink 2. DSP videosink v.s DSP audiosink 3. software videosink v.s DSP audiosink 4. software videosink v.s software audiosink How to sync a/v with encoded buffers? It really puzzled me. openmax can do it well for some cases, as it has a clock in itself. but by my understanding, basesink is used to sync decoded buffers. Appreicate for your clarifying. -- Benoit Fouet Purple Labs S.A. www.purplelabs.com ------------------------------------------------------------------------ - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Gstreamer-embedded mailing list Gst...@li... https://lists.sourceforge.net/lists/listinfo/gstreamer-embedded |
From: Felipe C. <fel...@no...> - 2008-03-04 08:21:39
|
On Tue, 2008-03-04 at 10:10 +0800, ext Zhang Yanlong-PBVM47 wrote: > Yes, I think it is not a simple things if integrate gstreamer with other > hardware, e.g dsp. > Maybe implement a set of API in dsp side and provide it to gstreamer is > a good solution. > > BRs > Zhang > > -----Original Message----- > From: gst...@li... > [mailto:gst...@li...] On Behalf Of > Zhao Liang-E3423C > Sent: Tuesday, March 04, 2008 10:05 AM > To: Benoit Fouet > Cc: gst...@li... > Subject: Re: [gst-embedded] How to use DSP in gstreamer > > > Hi, > > Zhao Liang-E3423C wrote: > > Zhao Liang-E3423C wrote: > > > >> Hi all, > >> > >> On embedded device, DSP is used widely, many DSPs have different > >> features, for example: > >> > >> 1. decoder > >> DSP is just a hardware decoder > >> 2. decoder + sink > >> DSP is a decoder plus sink, it can directly accept encoded data, and > >> then decode it and render pcm data into audio device directly. > >> > > > > creating a sink with right caps should be enough for this one... > > > > Zhao Liang: By my experience, it is not simple like just adding a new > > > caps. > > Just a example, how to handle preroll? Generally, dsp > need to do > > initialization when start to work, so what is the time to do this? > > another issue is seek, how does DSP handle seek? > > > > > > >this is generally the demuxer/parser's job to handle seek, do you have > examples where it is up to the decoder to do it ? > > As it is a sink for DSP although it includes decoder functions, sink > would handle preroll, in current preroll mechanism, it only allows sink > do once preroll virtual function, all other buffers are pushed into > queue, but some hardwares need more buffers to do initialization. > > Another issue is if the DSP is audio decoder + sink, how to handle clock > line increasing? Currently, we can use gstaudiosink to do it, > gstringbuffer can handle all clock line increasing, but some DSP can not > do it if no data is fed into, so the clock calculation is really a > issue. If no mechanism, rtsp streaming application will not handle > buffer delay or lost. > > > >> 3. A/V sync > >> DSP can do A/V sync internally or not. > >> > > > > then you can ask basesink not to synchronize flows. > > > > Zhao Liang: If DSP is just a video decoder and sink, how does DSP sync > > > with audio sink? maybe current basesink considers more software sink > > than hardware sink. > > > > > > >I was talking about your hardware handling A/V synchronization, not the > opposite. > >buffers that are handled by sink are timestamped (by demuxers for > >instance) and are rightly handled by basesink when synchronization has > to be done in framework, and not in HW. > > I understood your meaning, A/V sync is really done by basesink, but some > DSPs also have internal buffers to store buffers synced. and these > synced buffers are encoded buffers, not raw buffers. > There are 4 combinations: > 1. DSP videosink v.s software audiosink 2. DSP videosink v.s DSP > audiosink 3. software videosink v.s DSP audiosink 4. software > videosink v.s software audiosink When there's a decoder+sink, one approach is to map the hw clock to the GStreamer clock, that way synchronization is still handled by GStreamer transparently. However, I would like to see some numbers regarding the performance improvement when doing such thing (decoder+sink), I wonder if it's worth the trouble. Best regards. -- Felipe Contreras |
From: Zhao Liang-E. <E3...@mo...> - 2008-03-04 08:40:49
|
On Tue, 2008-03-04 at 10:10 +0800, ext Zhang Yanlong-PBVM47 wrote: > Yes, I think it is not a simple things if integrate gstreamer with > other hardware, e.g dsp. > Maybe implement a set of API in dsp side and provide it to gstreamer > is a good solution. > > BRs > Zhang > > -----Original Message----- > From: gst...@li... > [mailto:gst...@li...] On Behalf Of > Zhao Liang-E3423C > Sent: Tuesday, March 04, 2008 10:05 AM > To: Benoit Fouet > Cc: gst...@li... > Subject: Re: [gst-embedded] How to use DSP in gstreamer > > > Hi, > > Zhao Liang-E3423C wrote: > > Zhao Liang-E3423C wrote: > > > >> Hi all, > >> > >> On embedded device, DSP is used widely, many DSPs have different > >> features, for example: > >> > >> 1. decoder > >> DSP is just a hardware decoder > >> 2. decoder + sink > >> DSP is a decoder plus sink, it can directly accept encoded data, > >> and then decode it and render pcm data into audio device directly. > >> > > > > creating a sink with right caps should be enough for this one... > > > > Zhao Liang: By my experience, it is not simple like just adding a > > new > > > caps. > > Just a example, how to handle preroll? Generally, dsp > need to do > > initialization when start to work, so what is the time to do this? > > another issue is seek, how does DSP handle seek? > > > > > > >this is generally the demuxer/parser's job to handle seek, do you > >have > examples where it is up to the decoder to do it ? > > As it is a sink for DSP although it includes decoder functions, sink > would handle preroll, in current preroll mechanism, it only allows > sink do once preroll virtual function, all other buffers are pushed > into queue, but some hardwares need more buffers to do initialization. > > Another issue is if the DSP is audio decoder + sink, how to handle > clock line increasing? Currently, we can use gstaudiosink to do it, > gstringbuffer can handle all clock line increasing, but some DSP can > not do it if no data is fed into, so the clock calculation is really a > issue. If no mechanism, rtsp streaming application will not handle > buffer delay or lost. > > > >> 3. A/V sync > >> DSP can do A/V sync internally or not. > >> > > > > then you can ask basesink not to synchronize flows. > > > > Zhao Liang: If DSP is just a video decoder and sink, how does DSP > > sync > > > with audio sink? maybe current basesink considers more software sink > > than hardware sink. > > > > > > >I was talking about your hardware handling A/V synchronization, not > >the > opposite. > >buffers that are handled by sink are timestamped (by demuxers for > >instance) and are rightly handled by basesink when synchronization > >has > to be done in framework, and not in HW. > > I understood your meaning, A/V sync is really done by basesink, but > some DSPs also have internal buffers to store buffers synced. and > these synced buffers are encoded buffers, not raw buffers. > There are 4 combinations: > 1. DSP videosink v.s software audiosink 2. DSP videosink v.s DSP > audiosink 3. software videosink v.s DSP audiosink 4. software > videosink v.s software audiosink > When there's a decoder+sink, one approach is to map the hw clock to the GStreamer clock, that way synchronization is still handled by GStreamer transparently. > However, I would like to see some numbers regarding the performance improvement when doing such thing (decoder+sink), I wonder if it's worth the trouble. It is a good ideal to wrap hw clock to gstclock. For decoder+sink, sometimes it is restricted by HW, I saw nokia N800 using DSP gateway as audio sink, by my understanding, is it a decoder+sink? If you can share some experience on it, it will be appreciated. > Best regards. -- > Felipe Contreras ------------------------------------------------------------------------ - This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Gstreamer-embedded mailing list Gst...@li... https://lists.sourceforge.net/lists/listinfo/gstreamer-embedded |