[MPlayer-dev-eng] A better explanation of the A/V sync problem
Ross Finlayson
finlayson at live.com
Mon Mar 3 13:26:38 CET 2003
I was able to figure out more details about what was causing the A/V sync
problem. Basically, the problem is caused by the fact that UDP streams -
unlike TCP - have no control backchannel, so incoming UDP packets keep
arriving (to be buffered within the OS) at roughly the same rate,
regardless of the rate at which MPlayer reads them. Because MPlayer
current does A/V synchronization primarily by adjusting the rate at which
it reads audio and/or video (rather than by dropping audio or video
frames), it has difficulty maintaining synchronization for such streams.
This problem will occur for any audio/video session in which at least one
of the demuxers uses UDP. I was able to add a fix (described below) to my
RTP demuxing code to overcome the problem, but in the long term, I think it
would be better if the general MPlayer A/V sync code were to do this instead.
Once again, 'pausing' provides a good illustration of the problem. Suppose
that the user pauses an audio/video RTP session. Although MPlayer is not
reading any packets during the pause, packets continue to stream in over
the network, to be buffered by the OS's UDP socket buffer. (The size of
this OS buffer is configurable; I have set it to be quite large: 2 Mbytes
for the video socket, and 100 kBytes for the audio socket.) Nonetheless,
during a pause, these buffers will eventually fill up, after which any
subsequent arriving packets get 'dropped on the floor'. Usually, the video
socket's buffer fills up first. For illustration, suppose that, during the
pause, 10 seconds worth of video data got lost, but no audio data got lost.
After the pause, MPlayer continues reading audio and video packets once
again. Soon, it will reach the point at which video data was lost, at
which point it will see the video packets' RTP timestamps (and thus
presentation times) jump by 10 seconds. At this point, video is 10 seconds
ahead of audio! MPlayer tries to recover from this by reading from the
video demuxer at a slower rate than from the audio demuxer - but this just
has the effect of causing the OS's video socket buffer to fill up once
again! Although MPlayer begins to get back in sync, once the OS video
buffer fills up (and video packets get lost again), MPlayer will see
another large gap in the video data, which will cause it to get way out of
sync yet again. This cycle can continue indefinitely.
(I have also seen a similar situation occur - although less repeatably -
when video starts dropping behind audio.)
The obvious solution to this problem is to notice when the presentation
times on the two incoming streams start differing by more than a
threshold. When this happens, start discarding incoming packets from the
stream that is running behind, until its presentation times catch up to the
other stream. This is what I've now done in my RTP demuxer code. In the
'pausing' example described above, when the code notices that video has
jumped way ahead of audio, it will read and discard all following audio
packets until audio catches up again. I.e., if incoming data for one
stream gets dropped (by the OS) during the pause, then a corresponding
amount of data for the other stream will also get dropped (by MPlayer's RTP
demuxer) to compensate.
Within a day or so I will be submitting a patch to my RTP code that
includes this fix (plus a lot more goodies, such as MPEG-4
support). However, I think that in the longer term, it might be better if
the general MPlayer A/V sync code did this instead.
Ross.
More information about the MPlayer-dev-eng
mailing list