From: <wt...@ke...> - 2007-09-16 19:14:09
|
CVS Root: /cvs/gstreamer Module: gst-plugins-good Changes by: wtay Date: Sun Sep 16 2007 19:14:13 UTC Log message: * gst/rtp/README: Update README with the design for synchronisation rules of RTP on sender and receiver. Modified files: . : ChangeLog gst/rtp : README Links: http://freedesktop.org/cgi-bin/viewcvs.cgi/gstreamer/gst-plugins-good/ChangeLog.diff?r1=1.3092&r2=1.3093 http://freedesktop.org/cgi-bin/viewcvs.cgi/gstreamer/gst-plugins-good/gst/rtp/README.diff?r1=1.12&r2=1.13 ====Begin Diffs==== Index: ChangeLog =================================================================== RCS file: /cvs/gstreamer/gst-plugins-good/ChangeLog,v retrieving revision 1.3092 retrieving revision 1.3093 diff -u -d -r1.3092 -r1.3093 --- ChangeLog 14 Sep 2007 09:40:48 -0000 1.3092 +++ ChangeLog 16 Sep 2007 19:13:57 -0000 1.3093 @@ -1,3 +1,9 @@ +2007-09-16 Wim Taymans <wim...@gm...> + + * gst/rtp/README: + Update README with the design for synchronisation rules of RTP on + sender and receiver. 2007-09-14 Sebastian Dröge <sl...@ci...> * gst/wavparse/gstwavparse.c: (gst_wavparse_loop), Index: README RCS file: /cvs/gstreamer/gst-plugins-good/gst/rtp/README,v retrieving revision 1.12 retrieving revision 1.13 diff -u -d -r1.12 -r1.13 --- README 11 May 2007 15:04:38 -0000 1.12 +++ README 16 Sep 2007 19:13:58 -0000 1.13 @@ -24,10 +24,28 @@ * clock-rate: (int) [0 - MAXINT] The RTP clock rate. + encoding-name: (String) ANY + typically second part of the mime type. ex. MP4V-ES. only required if + payload type >= 96. Converted to upper case. + encoding-params: (String) ANY + extra encoding parameters (as in the SDP a=rtpmap: field). only required + if different from the default of the encoding-name. + Converted to lower-case. + ssrc: (uint) [0 - MAXINT] The ssrc value currently in use. (default = the SSRC of the first RTP packet) + clock-base: (uint) [0 - MAXINT] + The RTP time representing time npt-start. (default = rtptime of first RTP + packet). + seqnum-base: (uint) [0 - MAXINT] + The RTP sequence number representing the first rtp packet. When this + parameter is given, all sequence numbers below this seqnum should be + ignored. (default = seqnum of first RTP packet). npt-start: (uint64) [0 - MAXINT] The Normal Play Time for clock-base. This is the position in the stream and is between 0 and the duration of the stream. This value is expressed in @@ -37,10 +55,6 @@ The last position in the stream. This value is expressed in nanoseconds GstClockTime. (default = -1, stop unknown) - clock-base: (uint) [0 - MAXINT] - The RTP time representing time npt-start. (default = rtptime of first RTP - packet). - play-speed: (gdouble) [-MIN - MAX] The intended playback speed of the stream. The client is delivered data at the adjusted speed. The client should adjust its playback speed with this @@ -53,20 +67,6 @@ reporting and corresponds to the GStream applied-rate field in the NEWSEGMENT event. (default = 1.0) - seqnum-base: (uint) [0 - MAXINT] - The RTP sequence number representing the first rtp packet. When this - parameter is given, all sequence numbers below this seqnum should be - ignored. (default = seqnum of first RTP packet). - encoding-name: (String) ANY - typically second part of the mime type. ex. MP4V-ES. only required if - payload type >= 96. Converted to upper case. - encoding-params: (String) ANY - extra encoding parameters (as in the SDP a=rtpmap: field). only required - if different from the default of the encoding-name. - Converted to lower-case. - Optional parameters as key/value pairs, media type specific. The value type should be of type G_TYPE_STRING. The key is converted to lower-case. The value is left in its original case. @@ -117,6 +117,127 @@ time: <npt-start> +Timestamping +------------ +RTP in GStreamer uses a combination of the RTP timestamps and GStreamer buffer +timestamps to ensure proper synchronisation at the sender and the receiver end. +In RTP applications, the synchronisation is most complex at the receiver side. +At the sender side, the RTP timestamps are generated in the payloaders based on +GStreamer timestamps. At the receiver, GStreamer timestamps are reconstructed +from the RTP timestamps and the GStreamer timestamps in the jitterbuffer. This +process is explained in more detail below. += synchronisation at the sender +Individual streams at the sender are synchronised using GStreamer timestamps. +The payloader at the sender will convert the GStreamer timestamp into an RTP +timestamp using the following formula: + RTP = ((RT - RT-base) * clock-rate / GST_SECOND) + RTP-offset + RTP: the RTP timestamp for the stream. This value is truncated to + 32 bits. + RT: the GStreamer running time corresponding to the timestamp of the + packet to payload + RT-base: the GStreamer running time of the first packet encoded + clock-rate: the clock-rate of the stream + RTP-offset: a random RTP offset +The RTP timestamp corresponding to RT-base is the clock-base (see caps above). +In addition to setting an RTP timestamp in the RTP packet, the payloader is also +responsible for putting the GStreamer timestamp on the resulting output buffer. +This timestamp is used for further synchronisation at the sender pipeline, such +as for sending out the packet on the network. +Notice that the absolute timing information is lost; if the sender is sending +multiple streams, the RTP timestamps in the packets do not contain enough +information to synchronize them in the receiver. The receiver can however use +the RTP timestamps to reconstruct the timing of the stream as it was created by +the sender according to the sender's clock. +Because the payloaded packet contains both an RTP timestamp and a GStreamer +timestamp, it is possible for an RTP session manager to derive the relation +between the RTP and GST timestamps. This information is used by a session +manager to create SR reports. The NTP time in the report will contain the +running time converted to NTP time and the corresponding RTP timestamp. +Not that at the sender side, the RTP and GStreamer timestamp both increment at +the same rate, the sender rate. This rate depends on the global pipeline clock +of the sender. +Some pipelines to illustrate the process: + gst-launch v4l2src ! ffenc_h263p ! rtph263ppay ! udpsink + v4l2src puts a GStreamer timestamp on the video frames base on the current + running_time. The encoder encodes and passed the timestamp on. The payloader + generates an RTP timestamp using the above formula and puts it in the RTP + packet. It also copies the incomming GStreamer timestamp on the output RTP + packet. udpsink synchronizes on the gstreamer timestamp before pushing out the + packet. += synchronisation at the receiver +The receiver is responsible for timestamping the received RTP packet with the +running_time of the clock at the time the packet was received. This GStreamer +timestamp reflects the receiver rate and depends on the global pipeline clock of +the receiver. The gstreamer timestamp of the received RTP packet contains a +certain amount of jitter introduced by the network. +The most simple option for the receiver is to depayload the RTP packet and play +it back as soon as possible, this is with the timestamp when it was received +from the network. For the above sender pipeline this would be done with the +following pipeline: + gst-launch udpsrc caps="application/x-rtp, media=(string)video, + clock-rate=(int)90000, encoding-name=(string)H263-1998" ! rtph263pdepay ! + ffdec_h263 ! xvimagesink +It is important that the depayloader copies the incomming GStreamer timestamp +directly to the depayloaded output buffer. It should never attempt to perform +any logic with the RTP timestamp, this task is for the jitterbuffer as we will +see next. +The above pipeline does not attempt to deal with reordered packets or network +jitter, which could result in jerky playback in the case of high jitter or +corrupted video in the case of packet loss or reordering. This functionality is +performed by the gstrtpjitterbuffer in GStreamer. +The task of the gstrtpjitterbuffer element is to: + - deal with reordered packets based on the seqnum + - calculate the drift between the sender and receiver clocks using the + GStreamer timestamps (receiver clock rate) and RTP timestamps (sender clock + rate). +To deal with reordered packet, the jitterbuffer holds on to the received RTP +packets in a queue for a configurable amount of time, called the latency. +The jitterbuffer also eliminates network jitter and then tracks the drift +between the local clock (as expressed in the GStreamer timestamps) and the +remote clock (as expressed in the RTP timestamps). It will remove the jitter +and will apply the drift correction to the GStreamer timestamp before pushing +the buffer downstream. The result is that the depayloader receives a smoothed +GStreamer timestamp on the RTP packet, which is copied to the depayloaded data. +The following pipeline illustrates the sender with a jitterbuffer. + clock-rate=(int)90000, encoding-name=(string)H263-1998" ! + gstrtpjitterbuffer latency=100 ! rtph263pdepay ! ffdec_h263 ! xvimagesink +The latency property on the jitterbuffer controls the amount of delay (in +milliseconds) to apply to the outgoing packets. A higher latency will produce +smoother playback in networks with high jitter but cause a higher latency. +Choosing a good value for the latency is a tradeoff between the quality and +latency. The better the network, the lower the latency can be set. usage with UDP -------------- @@ -161,11 +282,12 @@ gst-launch-0.10 -v udpsrc caps="application/x-rtp, media=(string)video, payload=(int)96, clock-rate=(int)90000, encoding-name=(string)H263-1998, ssrc=(guint)527842345, clock-base=(guint)1150776941, seqnum-base=(guint)30982" - ! rtph263pdepay ! ffdec_h263 ! xvimagesink sync=false + ! rtph263pdepay ! ffdec_h263 ! xvimagesink - The receiver now displays an h263 image. Note that the sync parameter on - xvimagesink needs to be FALSE because we do not have an RTP session manager - that controls the synchronisation in this pipeline. + The receiver now displays an h263 image. Since there is no jitterbuffer in the + pipeline, frames will be displayed at the time when they are received. This can + result in jerky playback in the case of high network jitter or currupted video + when packets are dropped or reordered. Stream a quicktime file with mpeg4 video and AAC audio on port 5000 and port 5002. |