gstreamer-embedded

[gst-embedded] Video playback: frames memory consumption tuning

[Hugues F. <hug...@ya...>]

Using playbin to make a video playback will typically use this pipeline:
<decoder> ! queue ! <sink>
This make the decoding threading independent of the rendering threading, which is needed to let the rendering side synchronize on clock to display the frame of time N when the real time reach N, assuming that an amount of frames have been decoded by advance & have been put in the queue.
Gstreamer assumes that if queue is full, the source is blocked till sink has consumed some frames, so when the queue is full, decoder thread is stopped till rendering side as consumed a frame.
The problem is that by default, queue has 200 frames, so if the decoder is fast enough to decode faster than framerate, the queue will goes up to 200 frames !(in fact not because another 2s value limit the queue size).
By evidence, on embedded device, such a consumption is not acceptable, so we must tune the size of the queue.
The problem is that this tuning is at least codec-dependent (stream-dependent also):
1) MPEG4/H263
Decoding time is quite always the same for a given stream, so it is assumed that for a given fixed rendering framerate, decoder will never or rarely decode slower than framerate => small bufferization needed (2-3 frames).

2) H264
Decoding time has a big dispersion, some decoding "peaks" maybe there which makes a single decoding takes some framerate periods! => queue size must be larger (10-20 frames) to ensure bufferization. 
This also depends on:
- video dimensions, complexity of video, because this will decrease the decoding duration,
- framerate, because if frame rate is low regarding decoding duration, no pbm
- DPB of SPS header ? size of bufferization needed inside decoder to output 1 frame. Not sure if this must be care of.
=> so stream dependent

So we must be able to tune queue max-size-buffers=x, x depending on codec type (@ first approach) or stream infos (@second hand).

So the question are:
- Is anybody knows if someting is there in playbin allowing this ?
- If playbin modifications must be made, which type of gst mechanism can we use to get what is the video codec type & infos on video in order to tune the queue size ?


      _____________________________________________________________________________ 
Envoyez avec Yahoo! Mail. Une boite mail plus intelligente http://mail.yahoo.fr

Re: [gst-embedded] Video playback: frames memory consumption tuning

[Zhao Liang-E. <E3...@mo...>]

Huques,
 
I think there may be some methods.
1. not set queue size by frame number, but time of buffers.
2. change queue implementation to adapt to different codec ( not the best :) )
3. use GstController to control property dynamically.
 
Maybe someone else has better methods.
 
 
________________________________

From: gst...@li... [mailto:gst...@li...] On Behalf Of Hugues Fruchet
Sent: Thursday, March 13, 2008 8:42 PM
To: gst...@li...
Subject: [gst-embedded] Video playback: frames memory consumption tuning


Using playbin to make a video playback will typically use this pipeline:
<decoder> ! queue ! <sink>
This make the decoding threading independent of the rendering threading, which is needed to let the rendering side synchronize on clock to display the frame of time N when the real time reach N, assuming that an amount of frames have been decoded by advance & have been put in the queue.
Gstreamer assumes that if queue is full, the source is blocked till sink has consumed some frames, so when the queue is full, decoder thread is stopped till rendering side as consumed a frame.
The problem is that by default, queue has 200 frames, so if the decoder is fast enough to decode faster than framerate, the queue will goes up to 200 frames !(in fact not because another 2s value limit the queue size).
By evidence, on embedded device, such a consumption is not acceptable, so we must tune the size of the queue.
The problem is that this tuning is at least codec-dependent (stream-dependent also):
1) MPEG4/H263
Decoding time is quite always the same for a given stream, so it is assumed that for a given fixed rendering framerate, decoder will never or rarely decode slower than framerate => small bufferization needed (2-3 frames).
 
2) H264
Decoding time has a big dispersion, some decoding "peaks" maybe there which makes a single decoding takes some framerate periods! => queue size must be larger (10-20 frames) to ensure bufferization. 
This also depends on:
- video dimensions, complexity of video, because this will decrease the decoding duration,
- framerate, because if frame rate is low regarding decoding duration, no pbm
- DPB of SPS header ? size of bufferization needed inside decoder to output 1 frame. Not sure if this must be care of.
=> so stream dependent
 
So we must be able to tune queue max-size-buffers=x, x depending on codec type (@ first approach) or stream infos (@second hand).
 
So the question are:
- Is anybody knows if someting is there in playbin allowing this ?
- If playbin modifications must be made, which type of gst mechanism can we use to get what is the video codec type & infos on video in order to tune the queue size ?
 
 

________________________________

Envoyé avec Yahoo! Mail <http://us.rd.yahoo.com/mailuk/taglines/isp/control/*http://us.rd.yahoo.com/evt=52423/*http://fr.docs.yahoo.com/mail/overview/index.html> .
Une boite mail plus intelligente. 

Re: [gst-embedded] Video playback: frames memory consumption tuning

[Edward H. <edw...@co...>]

Hi all,

  There might be an even smarter/cleaner solution, but yet to be
implemented, by using the GST_EVENT_BUFFERSIZE event. It's initial goal
was to allow demuxers to inform downstream queues of how many
buffers/time/size they will need to buffer.

  If the handling was properly implemented in queue/multiqueue/queue2
you could then have elements that inform downstream elements of how much
minimum and/or maximum information they need to buffer.

  Edward

P.S. Unless this isn't clear enough, it is NOT YET implemented.

On Fri, 2008-03-14 at 09:47 +0800, Zhao Liang-E3423C wrote:
> Huques,
>  
> I think there may be some methods.
> 1. not set queue size by frame number, but time of buffers.
> 2. change queue implementation to adapt to different codec ( not the
> best :) )
> 3. use GstController to control property dynamically.
>  
> Maybe someone else has better methods.
>  
>  
> 
> ______________________________________________________________________
> From: gst...@li...
> [mailto:gst...@li...] On Behalf Of
> Hugues Fruchet
> Sent: Thursday, March 13, 2008 8:42 PM
> To: gst...@li...
> Subject: [gst-embedded] Video playback: frames memory consumption
> tuning
> 
> 
> 
> Using playbin to make a video playback will typically use this
> pipeline:
> <decoder> ! queue ! <sink>
> This make the decoding threading independent of the rendering
> threading, which is needed to let the rendering side synchronize on
> clock to display the frame of time N when the real time reach N,
> assuming that an amount of frames have been decoded by advance & have
> been put in the queue.
> Gstreamer assumes that if queue is full, the source is blocked till
> sink has consumed some frames, so when the queue is full, decoder
> thread is stopped till rendering side as consumed a frame.
> The problem is that by default, queue has 200 frames, so if the
> decoder is fast enough to decode faster than framerate, the queue will
> goes up to 200 frames !(in fact not because another 2s value limit the
> queue size).
> By evidence, on embedded device, such a consumption is not acceptable,
> so we must tune the size of the queue.
> The problem is that this tuning is at least codec-dependent
> (stream-dependent also):
> 1) MPEG4/H263
> Decoding time is quite always the same for a given stream, so it is
> assumed that for a given fixed rendering framerate, decoder will never
> or rarely decode slower than framerate => small bufferization needed
> (2-3 frames).
>  
> 2) H264
> Decoding time has a big dispersion, some decoding "peaks" maybe there
> which makes a single decoding takes some framerate periods! => queue
> size must be larger (10-20 frames) to ensure bufferization. 
> This also depends on:
> - video dimensions, complexity of video, because this will decrease
> the decoding duration,
> - framerate, because if frame rate is low regarding decoding duration,
> no pbm
> - DPB of SPS header ? size of bufferization needed inside decoder to
> output 1 frame. Not sure if this must be care of.
> => so stream dependent
>  
> So we must be able to tune queue max-size-buffers=x, x depending on
> codec type (@ first approach) or stream infos (@second hand).
>  
> So the question are:
> - Is anybody knows if someting is there in playbin allowing this ?
> - If playbin modifications must be made, which type of gst mechanism
> can we use to get what is the video codec type & infos on video in
> order to tune the queue size ?
>  
>  
> 
> 
> ______________________________________________________________________
> Envoyé avec Yahoo! Mail.
> Une boite mail plus intelligente. 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________ Gstreamer-embedded mailing list Gst...@li... https://lists.sourceforge.net/lists/listinfo/gstreamer-embedded
-- 
Edward Hervey <edw...@co...>
Collabora Multimedia