[FFmpeg-devel] rtp streaming x264+audio issues (and some ideas to fix them)

[Timo Teräs]

Hi all,

I tried to get a simple DV-cam to h264/aac RTP broadcast working.

Using:
  x264 0.84.1416 fcf70c2
  FFmpeg version SVN-r21640

Command line something like:
  dvgrab - | ffmpeg -threads 0 -f dv -i - \
        -an -vcodec libx264 -vpre hq -b 256K -bt 256K -s vga \
        -deinterlace -vglobal 1 -f rtp rtp://$IP:5004?pkt_size=1100 \
        -vn -acodec aac -ab 96K -ac 1 -flags +global_header \
        -f rtp rtp://$IP:5006?pkt_size=1100 -newaudio

Initially this was crashing right away (but streaming only video or
audio worked). I pinpointed the crash to ffmpeg.c:2119 where
  out_file->duration = in_file->duration;

in_file is NULL. I just did:
--- ffmpeg.c	(revision 21651)
+++ ffmpeg.c	(working copy)
@@ -2116,7 +2116,7 @@
         if (recording_time != INT64_MAX) {
             out_file->duration = recording_time / 1000000 * AV_TIME_BASE;
         } else {
-            out_file->duration = in_file->duration;
+            out_file->duration = in_file ? in_file->duration : 0;
         }
     }

And things started working to some degree. But I think the fix
is not right. The logic is wrong there, input_file[i] does not
necessarily map to output_file[i].

---

Now the second problem was that I was getting A-V sync problems.
It would appear that the RTP destinations do not synchronize each
other at all. And since the x264 encoder introduces ~2s delay
things get out of sync. (If I just recorded to a mp4 file or so
a-v sync was maintained).

Looking at the code it looks like the RTCP packet timestamps
are calculated relative to the first time they are sent which
is not really correct.

As a quick test, I did the following to ensure that the RTCP
packets were using same base time.

--- libavformat/rtpenc.c	(revision 21651)
+++ libavformat/rtpenc.c	(working copy)
@@ -91,7 +91,8 @@
     s->cur_timestamp = 0;
     s->ssrc = 0; /* FIXME: was random(), what should this be? */
     s->first_packet = 1;
-    s->first_rtcp_ntp_time = AV_NOPTS_VALUE;
+    //s->first_rtcp_ntp_time = AV_NOPTS_VALUE;
+    s->first_rtcp_ntp_time = ntp_time();
 
     max_packet_size = url_fget_max_packet_size(s1->pb);
     if (max_packet_size <= 12)

And indeed, after this vlc (with large enough file-cache) seemed
to play the live stream pretty well. However, it'd be better
if the RTP streams were synchronized so there isn't any manual
need to twiddle with the player cache sizes (and QT did not
like the stream in this case either).

What would be the proper way to fix the synchronization issue?
So that the RTP packets sent out are approximately (<1s difference)
from the original stream.

- Timo