Improving the audio callback. Removing audio glitches

From time to time I noticed that there are audio glitches at playback. These were usually in form of small gaps and sounded like a small jitter.

Audio callback

Audio in a PC works roughly like this: The soundcard tells the OS that it needs more data, maybe 10ms of audio data. And it needs those fast, maybe within the next 10ms. The OS then tells the audio application that it needs more data. That is where our code is called. This is called the audio callback. We must return a certain amount of data and we must return it fast. If we fail to be fast enough, the soundcard will not get the data fast enough and there will be a small gap / audio glitch.

There are many systems in between. There is the soundcard driver, then there is the OS audio layer (on MacOSX, that is CoreAudio). We use PortAudio as an additional small layer in our application, which supports CoreAudio and other audio interfaces.

At all steps, there are many parameters which can be tuned. First of all, you probably have the sample rate fixed (e.g. 48000 Hz), as well as the number of channels (2, stereo). Then, you tell the audio interface about the latency you want to allow. This basically gives you an upper bound about how much time you are allowed to spend in your audio callback. A normal value for the latency is maybe 10ms but note that our audio callback should return in noticeably less time.

Multithreading

The audio callback is usually called in its own thread. We must return fast, so we should do as less calculation as possible in it. In a normal music player, there is another thread which does the encoding from MP3 or whatever into the native format (PCM at 48kHz). Multithreaded programming is not easy when you need to pass data from one thread to another. Usually, this is made safe via locks. I.e. there is a common container, e.g. a list of raw data chunks, and there are locking mechanisms which ensure that only one thread at a time can access the list. When one thread accesses the data, it holds the lock. If another thread tries to access it then, it must wait for the lock.

Notice that this is bad for our audio callback! Any waiting is bad. It means that the OS must at least once switch to the other thread and the other thread must complete the access to the list and must release the lock and then the OS must finally switch back to our audio callback thread. This will often take too long.

Lock-free programming

The solution is to use only wait-free operations in the audio callback. This is a concept of lock-free programming. There are certain low-level operations you can do which have well-defined behavior when you do them on the same data from multiple threads at once, e.g. the operation compare-and-swap. Such operations are called atomic. Only some low-level data types such as int32 have such operations. With such base, you can build structures such as our list which don't use locks. This is even much more complicated than normal multithreaded programming! This is implemented now for the audio callback related code.

Realtime thread constraints

The OS still can be somewhat in a bad temper sometimes and give your audio callback thread only very little computation time because it thinks at some point that some other operation/application is more important and needs more CPU resources. This can still lead to a too delayed return of the audio callback. However, modern OS have some way to tell them that your audio callback thread is very important and always needs a certain amount of computation time, no matter what other heavy things you are doing on your PC at some point. This is called a realtime constraint. The audio callback sets its own thread constraint this way now. Again, there are a few parameters which can be tuned.


Another good article about all this can be read here.

The implementation of a linked list with certain wait-free and lock-free operations is here and there is a buffer structure based on it (hpp and cpp). All the code involved in the audio callback should be wait-free now!

The audio callback setup code is here. The realtime setup code is in setRealtime. The audio callback itself is paStreamCallback. The main important function it calls is player->readOutStream. The code of that function can be found at the somewhat messy file here (search for PlayerObject::readOutStream).

The 20140111-1-ea626be MacOSX release is the first binary release which contains all these new developments. All relevant code changes together can be seen here.

Posted by Albert Zeyer 2014-01-11 Labels: audio