Re: [Audacity-devel] The ALSA Midi timing puzzle

[Roger D. <rb...@cs...>]

I finally think I have a clear picture of the audio/midi synchronization
problem on Linux. PortAudio is telling us neither the DAC time of the
current callback nor the correct latency. I really struggled with what I
thought would be a quick fix because I didn't have a clear way to
represent what's going on or a good way to reason about it. I feel that
I wasted many days going in mental circles, and I spent more hours
trying to document things in a compact form that invites analysis. But
finally, here it is, ready for doxygen in AudioIO.cpp, and I should have
a finished implementation ready to test tonight and will report results.

Next, I have a backlog of questions from Paul about startup behavior,
underflow/overflow behavior, and some questions of my own about the
initial state of buffers. E.g. in the past, Audacity has been taking
multiple audio callbacks and outputting a full buffer of zeros (~125ms
or over 5K samples on Linux) before outputting the first actual sample
from any audio track.

If things work out, it remains to be seen when and how this information,
approach, and linux implementation should (or should not) be integrated
into the code. I wish I could say it's all running, it's perfect, and
does the right thing on OS X and Windows as well, but that's not the case.

-Roger

  \par The Big Picture
@verbatim

Sample
Time (in seconds, = total_sample_count / sample_rate)
  ^
  |                                             /         /
  |             y=x-mSystemTimeMinusAudioTime /         /
  |                                         /     #   /
  |                                       /         /
  |                                     /   # <- callbacks (#) showing
  |                                   /#        /   lots of timing jitter.
  |       top line is "full buffer" /         /     Some are later,
  |                     condition /         /       indicating buffer is
  |                             /         /         getting low. Plot
  |                           /     #   /           shows sample time
  |                         /    #    /             (based on how many
  |                       /    #    /               samples previously
  |                     /         /                 *written*) vs. real
  |                   / #       /                   time.
  |                 /<------->/ audio latency
  |               /#       v/
  |             /         / bottom line is "empty buffer"
  |           /   #     /      condition = DAC output time =
  |         /         /
  |       /      # <-- rapid callbacks as buffer is filled
  |     /         /
0 +...+---------#---------------------------------------------------->
  0 ^ |         |                                            real time
    | |         first callback time
    | mSystemMinusAudioTime
    |
    Probably the actual real times shown in this graph are very large
    in practice (> 350,000 sec.), so the X "origin" might be when
    the computer was booted or 1970 or something.


@endverbatim

  To estimate the true DAC time (needed to synchronize MIDI), we need
  a mapping from track time to DAC time. The estimate is the theoretical
  time of the full buffer (top diagonal line) + audio latency. To
  estimate the top diagonal line, we "draw" the line to be at least
  as high as any sample time corresponding to a callback (#), and we
  slowly lower the line in case the sample clock is slow or the system
  clock is fast, preventing the estimated line from drifting too far
  from the actual callback observations. The line is occasionally
  "bumped" up by new callback observations, but continuosly
  "lowered" at a very low rate.  All adjustment is accomplished
  by changing mSystemMinusAudioTime, shown here as the X-intercept.\n
    theoreticalFullBufferTime = realTime - mSystemMinusAudioTime\n
  To estimate audio latency, notice that the first callback happens on
  an empty buffer, but the buffer soon fills up. This will cause a rapid
  re-estimation of mSystemMinusAudioTime. (The first estimate of
  mSystemMinusAudioTime will simply be the real time of the first
  callback time.) By watching these changes, which happen within ms of
  starting, we can estimate the buffer size and thus audio latency.
  So, to map from track time to real time, we compute:\n
    DACoutputTime = trackTime + mSystemMinusAudioTime\n
  There are some additional details to avoid counting samples while
  paused or while waiting for initialization, MIDI latency, etc.