Re: [Mvpmc-devel] Audio Sync problems workaround

[Tom M. <tme...@gm...>]

Simon Hyde wrote:
> The "record" button instigates a brief video pause to allow the audio
> (which is running "behind" the video) to catch up. A glitch in the
> recorded stream usually causes video glitches, causing video frames
> to be dropped, so the audio ends up "ahead" of the video. Pressing
> the record button will only push the audio and video further out of
> sync by pausing the video to allow the audio to get even further 
> ahead.

Makes sense.

I've always operated with the understanding that hitting the record 
button only delays the video, and while I've encountered situations that 
started out out-of-sync and haven't been able to correct them with the 
record button, I've yet to observe a situation where sync was achieved 
and hitting the record button more pushed it out of sync. It's as if you 
can't hit the record button too much. I don't know if this is just 
coincidental, or if there is something in the code that would explain 
this, such as logic that prevents pushing the video out of sync if sync 
is established. (The situation feels sort of like a phase locked loop, 
where there is a small range above and below the target frequency where 
sync can be established, but outside of that range it looses it.)

Similarly, it also seems that hitting the record button once or twice 
has no perceptible effect. This could be because they delay is simply 
below perception, but I have a feeling that this is the result of 
another threshold. Until you've hit the button 4 or more times, the 
delay isn't enough to cause the hardware to resync.


> Perhaps we aught to offer a button to pause some audio to allow the
> video to catch up.

A common UI used by VCRs is to overload the channel up/down buttons as 
tracking control during playback. Seems we could do the same. Though 
obviously avoiding the need for manual controls at all would be ideal.

You mentioned that seeks now reset the STC, and that this has the effect 
of forcing the hardware to resync. Perhaps the record button should be 
repurposed to reset the STC. This may permit a manual resync after a 
video glitch without necessitating a skip.


> The audio is tagged with time code (a Presentation Time Stamp or PTS
> to be exact), and my code tries to keep this in sync. The problem is
> that there's a large hardware buffer for both the audio and video
> after the MVPMC software has sent it on to the MediaMVP hardware MPEG
> audio/video decoders.

Ah, thanks for the background.

Do you know if the MVP hardware attempts to do any synchronization on 
the two streams, or does it rely on the buffers being filled with data 
that is aligned?

The mere presence of the buffers implies that it likely is doing 
synchronization, though it is still conceivable that it just blindly 
pulls audio and video chunks from the buffers in parallel, assuming they 
are aligned. If it is synchronizing the streams at the far end of the 
buffer, it apparently isn't doing a very good job of it.

Is data flow through the buffers predictable? If we maintain sync at the 
"back end" of the buffers, does that guarantee that they remain in sync 
at the front end? (Assuming uncorrupted data.)

Can we bypass the buffers? Or do we only have access to the back-end? Do 
we have the ability to flush the buffers? Would buffering the data in 
mvpmc code, and flushing the buffers after each packet write kill the 
performance?


> The problem here is that we have no documentation for the
> MVP's hardware decoder (which is part of the IBM PowerPC processor on
> the MVP) and how to retrieve information from it.

I'd be willing to spend some time making phone calls and such to track 
down documentation, if that would help. Are there any notes somewhere on 
what has been dug up so far, and has anyone made contact with IBM?


> This would cause the JIT code...

JIT?


> It wouldn't suprise me if the same occured if a few video frames were
> lost. The only way around this is to reset the STC to 0 at which
> point it will resync to the video (which is what is now done whenever
> a seek occurs), but I'm not sure how you can detect that a problem
> has occured to force this STC reset, or indeed if this would really
> help matters much even then.
...
> I suppose you *could* keep an eye on the incoming MPEG video stream
> somewhere in the demuxer for glitches in PTS and do all sorts of
> things if this happened.

That sounds like a strategy worth exploring.

Still, one wonders what Hauppauge does to address this (assuming their 
video playback is more resilient to sync errors). Are they using the 
hardware more effectively, or did they resort to similar tricks in their 
software to manage the problem?

  -Tom