Audio distortion / artifacts when streaming to YouTube

[Aidan]

Hi Peter,

I've noticed that when we stream to YouTube live we get a lot of audio distortion / artifacts that we dont get when we stream to other destinations.

Here is a link of the audio problem we experience:

Audio Problem Video

(i've taken a screencast of the dvr playback because once the video has been processed into on demand the audio problem seems to disappear).

When we stream using the testsrc for video and audio there is no sign of the artifacts.

Currently Snowmix is running on a c4.xlarge instance in AWS. I've tried increasing the size of the instance as well as running it locally on a mac and I've got the same problem.

I've also tried running some of the demo ini's and combinations of video and audio sources without any further success.

I've included scripts that we run to stream to youtube as well as the ini for snowmix and an example of an input into snowmix. They are mostly based on help that you've given to other peope in the forum as well as your CDN script.

Any help would be very appreciated.

Thanks,

Aidan

controller.ini

# Basic initialization
system control port 9999
system geometry 1280 720 BGRA
system frame rate 25
system socket /tmp/mixer1
feed idle 0 1

# Necessary to prevent black screen for some reason
text string 0 Example

command create MyFunctions.tcl
  set vfeeds_to_overlay 0

  proc SetFeedToOverlay { args } {
    global vfeeds_to_overlay
    if {[llength $args]} { set vfeeds_to_overlay $args }
  }

  proc OverlayVFeed {} {
    global vfeeds_to_overlay
    if {[llength $vfeeds_to_overlay]} {
      snowmix parse "vfeed overlay $vfeeds_to_overlay"
    }
  }

  proc CreateFeed { id } {
    snowmix parse "feed add $id Feed $id"
    snowmix parse "feed geometry $id 1280 720"
    snowmix parse "feed live $id"
    snowmix parse "feed idle $id 100 frames/dead-bars.bgra"
    snowmix parse "feed socket $id /tmp/feed$id-control-pipe"

    snowmix parse "vfeed add $id VFeed $id"
    snowmix parse "vfeed source feed $id $id"
    snowmix parse "vfeed place rect $id 0 0 1280 720 0 0 0.0 1.0 1.0 1.0"
  }
command end

tcl exec MyFunctions.tcl

tcl eval CreateFeed 1
tcl eval CreateFeed 2
tcl eval CreateFeed 3
tcl eval CreateFeed 4
tcl eval CreateFeed 5
tcl eval CreateFeed 6
tcl eval CreateFeed 7
tcl eval CreateFeed 8
tcl eval CreateFeed 9
tcl eval CreateFeed 10
tcl eval CreateFeed 11
tcl eval CreateFeed 12

#load the slates 5 6 7 and 8
image load 5 images/overlay-slate.png
image load 6 images/overlay2-slate.png
image load 7 images/overlay3-slate.png
image load 8 images/overla4-slate.png

vfeed source image 5 5
vfeed source image 6 6
vfeed source image 7 7
vfeed source image 8 8

#load the bugs
image load 9 images/feed-9-bug.png
image load 10 images/feed-10-bug.png
image load 11 images/feed-11-strap.png

#audio feed verbose 1
audio feed add 1 Audio Feed #1
audio feed channels 1 2
audio feed rate 1 48000
audio feed format 1 16 signed
audio feed mute off 1
#audio feed volume 1 1
#audio feed monitor on 1

audio feed add 2 Audio Feed #2
audio feed channels 2 2
audio feed rate 2 48000
audio feed format 2 16 signed
audio feed mute on 2
#audio feed volume 2 1

audio feed add 3 Audio Feed #3
audio feed channels 3 2
audio feed rate 3 48000
audio feed format 3 16 signed
audio feed mute on 3
#audio feed volume 3 1

audio feed add 4 Audio Feed #4
audio feed channels 4 2
audio feed rate 4 48000
audio feed format 4 16 signed
audio feed mute on 4
#audio feed volume 4 1

#audio mixer verbose 1
audio mixer add 1 Audio Mixer #1
audio mixer channels 1 2
audio mixer rate 1 48000
#audio mixer format 1 16 signed
audio mixer mute off 1
#audio mixer volume 1 1
# audio mixer source (feed | mixer) <mixer id> <source id>
audio mixer source feed 1 1
audio mixer source feed 1 2
audio mixer source feed 1 3
audio mixer source feed 1 4
#audio mixer source volume 1 1 1
#audio mixer source volume 1 2 1
#audio mixer source volume 1 3 1
#audio mixer source volume 1 4 1
audio mixer source mute off 1 1
audio mixer source mute off 1 2
audio mixer source mute off 1 3
audio mixer source mute off 1 4
##audio mixer drop source 1 2
##audio mixer volume 1 1 1
##audio mixer volume 1 2 1

audio sink add 0 Dummy Sink
audio sink channels 0 1
audio sink rate 0 48000
audio sink format 0 16 signed
audio sink source mixer 0 1

#audio sink verbose 1
audio sink add 1 Audio Sink #1
audio sink channels 1 2
audio sink rate 1 48000
audio sink format 1 16 signed
audio sink mute off 1
#audio sink source feed 1 4
audio sink source mixer 1 1
#audio sink queue
#audio sink queue maxdelay <sink id> <max delay in ms>
#audio sink queue dropbuffer <sink id> <buffers>
#audio sink queue maxdelay 1 120
#audio sink file 1 pmm.pipe

audio feed verbose 1
audio mixer verbose 1
audio sink verbose 1
audio sink start 0
#audio sink start 1
audio mixer start 1

    command create Show
      tcl eval OverlayVFeed
      image overlay all
      text overlay all
      loop
    command end
    overlay finish Show

#this command makes the overlays work
overlay finish Show
#this sets feed1 as running
tcl eval SetFeedToOverlay 1

#create message logs
    command create MessageLog.tcl
      snowmix parse "message Tcl script begins\n"
      proc SendMessage {str} {
        snowmix parse "message $str : SendMessage was called\n"
        return
      }
      snowmix parse "message Tcl script ends\n"
    command end
    tcl exec MessageLog.tcl
    ...
    tcl eval SendMessage Hello

input1.sh:

while true ; do
rm -f /tmp/feed1-control-pipe
(echo 'audio feed ctr isaudio 1' ; gst-launch-1.0 -v filesrc location=/Users/aidanbarrett/Developer/Snowmix-0.5.1/test/bbb.mp4 !\
    queue !\
    decodebin name=decoder !\
    videoconvert  !\
    videorate !\
    videoscale !\
    videoconvert !\
    video/x-raw,format=BGRA,pixel-aspect-ratio=1/1,interlace-mode=progressive, width=1280, height=720 !\
    shmsink socket-path=/tmp/feed1-control-pipe shm-size=95846400 wait-for-connection=1 sync=true decoder. !\
    queue !\
    audioconvert !\
    audioresample !\
    audio/x-raw,format=S16LE,layout=interleaved, rate=48000, channels=2 !\
    fdsink fd=3 sync=true 3>&1 1>&2 ) | nc 127.0.0.1 9999

    sleep 2
done
exit

youtube.sh

   #!/bin/bash
    if [ $# != 1 ] ; then
      echo "Parameter error"
      echo "Usage : $0 you | you1 | you2 | aka | akamai | ustream"
      exit
    fi

    ################### YouTube Settings ###################
    # These data are available to you when you create a Live event in YouTube
    # First Camera - INSERT YOUR OWN DATA
    youtube_auth1='KEY_GOES_HERE'
    # Second Camera - Optional
    youtube_auth2=''
    youtube_app='live2'
    serverurl='rtmp://a.rtmp.youtube.com/'$youtube_app
    ################### UStream Settings ###################
    # RTMP URL from your UStream Account : See www.ustream.tv -> Channel -> Remote
    rtmpurl='INSERT_YOUR_USTREAM_RTMP_URL_HERE'
    # This is your Stream Key : See www.ustream.tv -> Channel -> Remote
    streamkey='INSERT_YOUR_STREAM_KEY_HERE'
    ################### Twitch Settings ###################
    twitch_server=live.twitch.tv
    twitch_streamkey='INSERT_YOUR_STREAM_KEY_HERE'
    # twitch_streamkey=live_12345678_6asdk3khhewrkhqe4k32AswlH6hrwd
    ################### Akamai Settings ###################
    akamai_server='INSERT_YOUR_AKAMAI_SERVER_NAME_HERE'
    akamai_user='INSERT_YOUR_AKAMAI_USER_NAME_HERE'
    akamai_pass='INSERT_YOUR_AKAMAI_PASSWORD_NAME_HERE'
    ########################################################
    # You can change the settings below to suit your needs
    ###################### Settings ########################
    width=1280
    height=720
    audiorate=48000
    channels=2
    framerate='25/1'
    vbitrate=4000
    abitrate=128000
    GST_DEBUG="--gst-debug=flvmux:0,rtmpsink:0"
    ###################### Settings ########################
    ########################################################
    # THe following settings should not be changed
    h264_level=4.1
    h264_profile=main
    h264_bframes=0
    keyint=`echo "2 * $framerate" |bc`
    datarate=`echo "$vbitrate + $abitrate / 1000" |bc`
    flashver='FME/3.0%20(compatible;%20FMSc%201.0)'
    akamai_flashver="flashver=FMLE/3.0(compatible;FMSc/1.0) playpath=I4Ckpath_12@44448"
    ########################################################

    # This will detect gstreamer-1.0 over gstreamer-0.10
    gstlaunch=`which gst-launch-1.0`
    if [ X$gstlaunch != X ] ; then
      VIDEOCONVERT=videoconvert
      VIDEO='video/x-raw, format=(string)BGRA, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive'
      AUDIO=audio/x-raw
      # VIDEO=video/x-raw-yuv
      vfid=string
      afid="format=(string)S16LE, "
    else
      gstlaunch=`which gst-launch-0.10`
      if [ X$gstlaunch != X ] ; then
        VIDEOCONVERT=ffmpegcolorspace
        VIDEO='video/x-raw-rgb, bpp=(int)32, depth=(int)32, endianness=(int)4321, red_mask=(int)65280, green_mask=(int)16711680, blue_mask=(int)-16777216, pixel-aspect-ratio=(fraction)1/1, interlaced=(boolean)false'
        AUDIO=audio/x-raw-int
        vfid=fourcc
        afid=""
      else
        echo "Could not find gst-launch-1.0 or gst-launch-0.10. Stopping"
        exit
      fi
    fi

    case $1 in
      you|you1|you2|youtube|youtube1|youtube2)
            if [ $1 = you2 -o $1 = youtube2 ] ; then
              auth="$youtube_auth2"
            else
              auth="$youtube_auth1"
            fi
            if [ X$auth = X ] ; then
              echo "auth was not set YouTube"
              exit 1
            fi
            ENCAUDIOFORMAT='aacparse ! audio/mpeg,mpegversion=4,stream-format=raw '
            videoencoder="x264enc bitrate=$vbitrate key-int-max=$keyint bframes=$h264_bframes byte-stream=false aud=true tune=zerolatency"
            audioencoder="faac bitrate=$abitrate"
            location="rtmp://a.rtmp.youtube.com/live2/axv3-v7jq-c8pu-1s3k"
            ;;
      ustream)
            videoencoder="x264enc bitrate=$vbitrate bframes=0"
            #ENCAUDIOFORMAT='audio/mpeg'
            ENCAUDIOFORMAT=mpegaudioparse
            abitrate=`echo "$abitrate / 1000" | bc`
            audioencoder="lamemp3enc bitrate=$abitrate ! mpegaudioparse"
            location="$rtmpurl/$streamkey live=1 flashver=$flashver"
            ;;
      twitch)
            if [ $vbitrate -gt 3000] ; then vbitrate=3000 ; fi
            videoencoder="x264enc bitrate=$vbitrate bframes=0"
            ENCAUDIOFORMAT=mpegaudioparse
            abitrate=`echo "$abitrate / 1000" | bc`
            audioencoder="lamemp3enc bitrate=$abitrate ! mpegaudioparse"
            location="rtmp://$twitch_server/app/$twitch_streamkey live=1 flashver=$flashver"
            ;;
      aka | akamai)
            videoencoder="x264enc bitrate=$vbitrate bframes=0"
            audioencoder="faac bitrate=$abitrate"
            ENCAUDIOFORMAT='aacparse ! audio/mpeg,mpegversion=4,stream-format=raw'
            stream_key="live=true pubUser=$akamai_user pubPasswd=$akamai_pass"
            location="rtmp://$akamai_server/EntryPoint $stream_key $akamai_flashver"
            ;;
      *)    echo 'Use youtube, akamai or ustream'
            exit
    esac

    ENCVIDEOFORMAT='h264parse ! video/x-h264,level=(string)'$h264_level',profile='$h264_profile
    VIDEOFORMAT=$VIDEO', framerate='$framerate', width='$width', height='$height
    AUDIOFORMAT=$AUDIO', '$afid' endianness=(int)1234, signed=(boolean)true, width=(int)16, depth=(int)16, rate=(int)'$audiorate', channels=(int)'$channels
    TIMEOLPARMS='halignment=left valignment=bottom text="Stream time:" shaded-background=true'
    VIDEOSRC="videotestsrc pattern=0 is-live=true"
    AUDIOSRC="audiotestsrc is-live=true"

   # ctrsocket must match system socket in running Snowmix
    ctrsocket=/tmp/mixer1
    # audiosink must match audio sink in running SNowmix
    audiosink=1
    VIDEOSRC='shmsrc socket-path='$ctrsocket' do-timestamp=true is-live=true'
    AUDIOSRC="fdsrc fd=0 do-timestamp=true"

    ( echo "audio sink ctr isaudio $audiosink" ; sleep 100000000 ) | \
    nc 127.0.0.1 9999      | \
    (head -1
     $gstlaunch -v                                   \
            $VIDEOSRC                               !\
            $VIDEOFORMAT                            !\
            queue                                   !\
            $VIDEOCONVERT                           !\
            $videoencoder                           !\
            $ENCVIDEOFORMAT                         !\
            queue                                   !\
            mux. $AUDIOSRC                          !\
            $AUDIOFORMAT                            !\
            queue                                   !\
            $audioencoder                           !\
            $ENCAUDIOFORMAT                         !\
            queue                                   !\
            flvmux streamable=true name=mux         !\
            queue                                   !\
            rtmpsink location="rtmp:/YOUTUBE_URL"
    )

[Peter Maersk-Moller]

Hi Aiden.

Try changing your samplerate to 44100 everywhere instead of 48000. The module flvmux does not handle 48000 Hz samplerate. Let me know if this does not help and I'll take a look at your setup. Make sure I have all the correct versions of ini files and scripts you are using.

Regards
Peter Maersk-Moller

[Aidan]

Hi Peter,
Thanks for the quick reply. I switched over the sample rate to 44100 and I've still got that same issue. I've attached the files I'm using in my setup below.
Thank you!

[Peter Maersk-Moller]

Hi Aidan.

I'll take a look at it and see what I can find out. However expect a few days. Time is a bit tight the next 2-3 days. Can you confirm that you are using faac? What version of GStreamer are you using (gst-inspect-1.0 --version) ?

Regards
Peter

[Aidan]

Thanks Peter, I really appreciate your help with this.
I can confirm that I am using faac for the audio encoding.
I'm using GStreamer 1.10.2 on my mac and GStreamer 1.9.90 on ubuntu.

I don't know if this helps but I ran ffprobe on the output to see if I could find anything and when I include audio from snowmix I get the following error messages:

flv @ 0x7fb6ac002a00] Non-increasing DTS in stream 1: packet 3 with DTS 844635, packet 4 with DTS 844635
[flv @ 0x7fb6ac002a00] Non-increasing DTS in stream 1: packet 5 with DTS 844678, packet 6 with DTS 844678
[flv @ 0x7fb6ac002a00] Non-increasing DTS in stream 1: packet 9 with DTS 844769, packet 10 with DTS 844769
[flv @ 0x7fb6ac002a00] Non-increasing DTS in stream 1: packet 11 with DTS 844812, packet 12 with DTS 844812
[flv @ 0x7fb6ac002a00] Non-increasing DTS in stream 1: packet 13 with DTS 844855, packet 14 with DTS 844855
[flv @ 0x7fb6ac002a00] Non-increasing DTS in stream 1: packet 16 with DTS 844919, packet 17 with DTS 844919
[flv @ 0x7fb6ac002a00] Non-increasing DTS in stream 1: packet 18 with DTS 844961, packet 19 with DTS 844961
[flv @ 0x7fb6ac002a00] Non-increasing DTS in stream 1: packet 20 with DTS 845004, packet 21 with DTS 845004

I tried this with the testsrc using the same encode and I don't receive those errors.

[Peter Maersk-Moller]

audiotestsrc is different from fdsrc because its timstamps starts at zero, fdsrc's timestamp does not. This affects several modules in GStreamer in (for me sometimes) unpredictable ways.

Couple of things to try while I look into it. Try add a

    queue ! audiorate

between fdsrc and and the queue element before faac. Worth a try. You can also try adding with and without audiorate the audioparse element.

    audioparse use-sinkcaps=true

regards
Peter

[Aidan]

Hi Peter,

That seems to have done the trick!
Adding the queue and audiorate elements seems to have fixed it straight away.

Thank you very much for your help!

[Peter Maersk-Moller]

Hi Aidan.

Glad to hear. I suspect the problem relates to the irregularity of the timestamps that the samples received by fdsrc gets. You will need samples to be timestamped, but apparently, the flvmux (and the receiving other end) can't handle irregular timestamps eventhough that the samples receievd over 1-2 frame periods ought to be pretty constant. The audiorate element will even out any irreguarity. However sometimes I have experienced that the audiorate element eventually stalls the pipeline. Any. Nice to know it works for you.

Regards
Peter

[Aidan]

Hi Peter,

With regard to the pipeline stalling I have seen the following messages flood into snowmix on random occasions:

Frame 1812 - audio sink 1 would block
Frame 1813 - audio sink 1 would block
Frame 1814 - audio sink 1 would block
Frame 1815 - audio sink 1 would block
Frame 1816 - audio sink 1 would block

When they appear the output seems to have stalled and I need to restart it. This has happened a couple of times since adding in the audiorate element and but it was also happening randomly before that was added in.

It doesn't seem to happen based on an event like starting or changing a feed. So i havent been able to replicate the behavior based on a particular action or workflow.

[Alex Vorona]

I have similar audio problems with YT streaming, timestamp of audio samples jumps back and forward according to audiorate filter with silence=false, which produces a lot of dropped samples and audio distortion due to this.
audiorate with 10 times increased from default tolerance minimized this issue.

audiorate tolerance=400000000

I suppose it just rewrites audio timestamps instead of drop samples with incorrect timestamps
audio mixer verbose 3 shows same problem with lacking samples as at https://sourceforge.net/p/snowmix/discussion/Snowmix_Support_Forum/thread/ab97b94a/ , I'm not sure if it's related to audio samples timestamp jumps.

[Peter Maersk-Moller]

Hi Alex.

Snowmix audio input require a steady reliable stream of raw audio samples. If you feed it too few, Snowmix will add samples of silence, which can will produce clicks mos of the time. If you feed it too many too quickly, there is a way to automatically drop samples above a threshold.

GStreamer has a number of methods to generate a steady stream of audio samples so adding a feature in Snowmix to deal with streams with audio time stamp issues is unnecessary and the wrong place to do it.

Regards
Peter MM

[Alex Vorona]

Hi Peter,

I tried to use audiorate gstreamer filter on snowmix inputs, it changes nothing and with silence=false doesn't produce warnings. So I suppose input streams are correct. I also tried other inputs, such as different files, and problem is still here . Here are steps to reproduce this issue

• start snowmix 0.5.1 with attached config test.ini
• add "audiorate silent=false !\" line to av_output2tcp_server after, for example, audioparse line
• start av_output2tcp_server
• start any inputs to 1'st snowmix feed with audio, my were "rtmp2feed 1 1" from camera and "input2feed 1 1" with different mp4 files
• see audiorate add/drop messages on av_output2tcp_server console, test with "nc 127.1 5010|mplayer - ", sound is choppy
• stop snowmix input, mixer still works as configured, and audiorate is producing adds and drops on audio despite of no snowmix audio inputs.
• stop av_output2tcp_server

av_output2tcp_server console and snowmix console with "audio mixer verbose 3" also attached

Of course it may be a docker-related or VM-related issue, unfortunately I cannot test on bare HW right now, but I got same problem in docker on my Mac and in docker on my KVM virtual machine.

Let me know if You need any additional info.

[Peter Maersk-Moller]

Hi Alex.

I'm not sure I understand all your issues. However a couple of remarks.

1) You are using an output pipeline that will stall if it does not receive both audio and video.
2) Your audio pipeline will not flow before an audio source connects to audio feed. Please read the section 'Starting audio mixers' on this page https://sourceforge.net/p/snowmix/wiki/Audio/

SOme of your problems will go away when you add these lines before starting audio mixer 1.

    audio sink add 0 Dummy
    audio sink channels 0 2
    audio sink rate 0 44100
    audio sink format 0 16 signed
    audio sink source mixer 0 1
    audio sink start 0

(Audio sink 0 is special.)

When you start the audio mixer 1 you will see it enters the state RUNNING and as soon as you connects the output pipeline you will receive audio samples, in your case 44100x2 samples per second from audio sink 1.

Now taking this as a starting point, what kind of issues are you now seeing, or rather hearing and how do you detect it?

Regards
Peter

[Alex Vorona]

Hi Peter,

I added sink 0 to my test config. Now output pipeline starts immediately. Issue I'm still having is choppy sound with YT streaming. Here is YT streaming screen capturing including streaming sound played through speakers and going into microphone https://vimeo.com/215792116 .
Problem detection without involving YT is still the same

• start snowmix 0.5.1 with attached config test.ini
• add "audiorate silent=false !\" line to av_output2tcp_server after, for example, audioparse line
• start av_output2tcp_server, add/drop messages from audiorate appear on console, I suppose due to wrong timestamps.
• start some inputs, for example "input2feed 1 1" with snowmix stock Bunny movie.
• start some player to play snowmix stream from tcp, sound is choppy same way as on YT streaming.

Without audiorate filter at av_output2tcp_server there is no sound distortion in local players, but in this case I also noticed ffmpeg timestamp warnings during stream recording
ffmpeg -i tcp://192.168.1.225:5050 -c:v copy -c:a aac -f flv -y 1.flv
...
[aac @ 0x7fb19c812c00] Queue input is backward in time. Last message repeated 159 times

I suppose YT streaming uses similar to gstreamer's audiorate sound filter, that's why YT streaming sound is choppy.

Let me know if You need any additional info.

[Peter Maersk-Moller]

Hi Alex.

Please post your complete pipeline script you use to send mixed AV stream to YT so I can recreate your setup.

For local player you can also use this:

    gst-launch-1.0 -v tcpclientsrc host=127.0.0.1 port=5010 ! decodebin name=decoder ! autovideosink decoder. ! autoaudiosink

The audiorate does not always do what I expect it to do.egards

Regards
Peter

[Alex Vorona]

Hi Peter,

I used youtube.sh, posted in this thread https://sourceforge.net/p/snowmix/discussion/Snowmix_Support_Forum/thread/8de359ec/#7e10/5a51/3ab1
my gst-inspect just in case http://termbin.com/55te
gst-inspect-1.0 version 1.10.2 GStreamer 1.10.2

Thanks for Your help,
Alex