[FFmpeg-devel] [RFC] Synchronize the start times of multiple RTP streams

[Martin Storsjö]

On Fri, 2 Apr 2010, Martin Storsj? wrote:

> On Fri, 2 Apr 2010, Ronald S. Bultje wrote:
> 
> > I'm starting to believe I was wrong here, also because my earlier
> > patch broken RTSP seeking. utils.c works based on
> > AVStream->start_time, rebasing it to a global timebase, and then
> > setting AVFormatContext->start_time to that. We set (I think validly)
> > these AVStream->start_time values to the range: value in the SDP,
> > which has no relation to RTP header timestamps. :-(. That's why it all
> > doesn't work the way it should.
> > 
> > I think we should revive your patch, with some fixes:
> > - we should test/verify it fixes seeking (in ffplay)
> > - I'm not quite sure why you do this under if (ret < 0), I think that
> > if can be removed, and in fact I'm wondering if you should only use it
> > if ret == 0/1 (?)
> 
> The reason for doing this only in ret < 0 is that receiving a RTCP packet 
> makes the parse routine return < 0 - and only if we received a RTCP 
> packet, the timestamps may have been initialized, so it isn't necessary to 
> check for the case when the first sender report was received otherwise.
> 
> > - for many streams, the first RTCP packet only comes after a bunch of
> > RTP packets. We want to make sure that after parsing, when they can be
> > assigned a timestamp, this doesn't get < 0
> 
> I'm not sure I understand what you're referring to at the end of this 
> sentence. The packets that have been received before the first RTCP have 
> been passed out from the RTSP code already and can't be updated anymore, 
> right?
> 
> As Luca pointed out, RTSP also passes a response header RTP-Info, in 
> response to PLAY, that actually gives the start RTP timestamp for all 
> streams. (This only is given for non-live, seekable streams, though.)
> 
> So one way to get proper timestamps for all packets from the beginning, 
> would be to use the RTP-Info header, if available. According to the RFC, 
> one should still use the later RTCPs for resyncing and taking potential 
> drift into account.
> 
> So, the complete solution perhaps would be to initially parse RTP-Info, if 
> available, use that as initial sync point for timestamps in rtpdec - then 
> when the first RTCP is received, sync that to the previously used sync 
> origin, and after that only use RTCP. I can try to cook up some patch that 
> does this. But except for using the RTP-Info if available, I think the 
> basic solution boils down to more or less my previous patch.
> 
> As for seeking actually working - with the patch I sent earlier, it 
> somewhat seems to work. Seeking forwards seems to work quite ok, but 
> seeking backwards doesn't really work that well at all.

Scratch that, seeking both forwards and backwards seems to work quite ok.

But for some really weird reason, the framerate when playing rtsp with 
ffplay is really really choppy, sometimes after seeking, sometimes 
immediately at the start, usually it starts working well after some 
seconds.

One would instinctively think that this is due to lack of RTCP, but it 
does happen even at the beginning, before any seeking has taken place, and 
when playing streams served from disk by the rtsp server (where it sends 
out RTCPs directly at the start).


After seeking, the timestamps of packets get way off, though, until the 
next RTCP sync. E.g., if watching a clip with a range from 0 to 70 sec. At 
10 seconds in the stream, I choose to seek to 60 sec. Then the timestamps 
of newly received packets are around ~60 sec (since the RTP timestamps are 
corresponding to the place in the video), but after the next RTCP packet, 
the timestamps of packets get back to around 15 sec (that is, the time 
since the first RTCP packet). The same happens when seeking backwards, 
too.

So, when seeking in RTSP, what should the timestamps of packets be? 
Corresponding to the npt (normal play time) of the actual video, or 
corresponding to the real time of the RTSP session (i.e., monotonically 
increasing during the session, despite seeking back and forth)?

Either way, I think one may want to reset the RTCP timestamp sync in one 
way or another after seeking.

// Martin