[FFmpeg-devel] [RFC] Synchronize the start times of multiple RTP streams

[Martin Storsjö]

On Sat, 3 Apr 2010, Martin Storsj? wrote:

> On Fri, 2 Apr 2010, Martin Storsj? wrote:
> 
> > On Fri, 2 Apr 2010, Ronald S. Bultje wrote:
> > 
> > > I'm starting to believe I was wrong here, also because my earlier
> > > patch broken RTSP seeking. utils.c works based on
> > > AVStream->start_time, rebasing it to a global timebase, and then
> > > setting AVFormatContext->start_time to that. We set (I think validly)
> > > these AVStream->start_time values to the range: value in the SDP,
> > > which has no relation to RTP header timestamps. :-(. That's why it all
> > > doesn't work the way it should.
> > > 
> > > I think we should revive your patch, with some fixes:
> > > - we should test/verify it fixes seeking (in ffplay)
> > > - I'm not quite sure why you do this under if (ret < 0), I think that
> > > if can be removed, and in fact I'm wondering if you should only use it
> > > if ret == 0/1 (?)
> > 
> > The reason for doing this only in ret < 0 is that receiving a RTCP packet 
> > makes the parse routine return < 0 - and only if we received a RTCP 
> > packet, the timestamps may have been initialized, so it isn't necessary to 
> > check for the case when the first sender report was received otherwise.
> > 
> > > - for many streams, the first RTCP packet only comes after a bunch of
> > > RTP packets. We want to make sure that after parsing, when they can be
> > > assigned a timestamp, this doesn't get < 0
> > 
> > I'm not sure I understand what you're referring to at the end of this 
> > sentence. The packets that have been received before the first RTCP have 
> > been passed out from the RTSP code already and can't be updated anymore, 
> > right?
> > 
> > As Luca pointed out, RTSP also passes a response header RTP-Info, in 
> > response to PLAY, that actually gives the start RTP timestamp for all 
> > streams. (This only is given for non-live, seekable streams, though.)
> > 
> > So one way to get proper timestamps for all packets from the beginning, 
> > would be to use the RTP-Info header, if available. According to the RFC, 
> > one should still use the later RTCPs for resyncing and taking potential 
> > drift into account.
> > 
> > So, the complete solution perhaps would be to initially parse RTP-Info, if 
> > available, use that as initial sync point for timestamps in rtpdec - then 
> > when the first RTCP is received, sync that to the previously used sync 
> > origin, and after that only use RTCP. I can try to cook up some patch that 
> > does this. But except for using the RTP-Info if available, I think the 
> > basic solution boils down to more or less my previous patch.
> > 
> > As for seeking actually working - with the patch I sent earlier, it 
> > somewhat seems to work. Seeking forwards seems to work quite ok, but 
> > seeking backwards doesn't really work that well at all.
> 
> Scratch that, seeking both forwards and backwards seems to work quite ok.
> 
> But for some really weird reason, the framerate when playing rtsp with 
> ffplay is really really choppy, sometimes after seeking, sometimes 
> immediately at the start, usually it starts working well after some 
> seconds.
> 
> One would instinctively think that this is due to lack of RTCP, but it 
> does happen even at the beginning, before any seeking has taken place, and 
> when playing streams served from disk by the rtsp server (where it sends 
> out RTCPs directly at the start).
> 
> 
> After seeking, the timestamps of packets get way off, though, until the 
> next RTCP sync. E.g., if watching a clip with a range from 0 to 70 sec. At 
> 10 seconds in the stream, I choose to seek to 60 sec. Then the timestamps 
> of newly received packets are around ~60 sec (since the RTP timestamps are 
> corresponding to the place in the video), but after the next RTCP packet, 
> the timestamps of packets get back to around 15 sec (that is, the time 
> since the first RTCP packet). The same happens when seeking backwards, 
> too.
> 
> So, when seeking in RTSP, what should the timestamps of packets be? 
> Corresponding to the npt (normal play time) of the actual video, or 
> corresponding to the real time of the RTSP session (i.e., monotonically 
> increasing during the session, despite seeking back and forth)?

If they should be corresponding to the original time stamps of the seeked 
to position in the video, things should be solved by the attached patch 
series (which still may need some brushing up, it's mainly a RFC series 
still).

With this series, we get proper time stamps for all packets, both from the 
start and after seeking. Seeking seems to work very well with this.

The patches in the series are more or less incremental, as in, each give a 
small enhancement in behaviour - the more of them you use, the better 
things behave. :-)

// Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Revert-svn-rev-21857-readd-first_rtcp_ntp_time-in-RT.patch
Type: text/x-diff
Size: 4042 bytes
Desc: 
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20100403/8cc247f6/attachment.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-Reset-RTCP-timestamps-after-seeking-add-range-start-.patch
Type: text/x-diff
Size: 3092 bytes
Desc: 
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20100403/8cc247f6/attachment-0001.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-Parse-RTP-Info-headers.patch
Type: text/x-diff
Size: 3472 bytes
Desc: 
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20100403/8cc247f6/attachment-0002.patch>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0004-Use-the-currently-unused-base_timestamp-for-setting-.patch
Type: text/x-diff
Size: 2952 bytes
Desc: 
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20100403/8cc247f6/attachment-0003.patch>