[FFmpeg-user] Understanding Latency Delays in FFmpeg

[Michael Casadevall]

I'm currently working on building a low-latency screen capture
application using FFmpeg, and VP8. Unfortunately, I've run into a
latency issue that I'm trying to resolve, and hoping someone can help
provide some insight into x11grab. Ideally, my target is <1 sec
latency, though decreasing that to 500ms or less would be ideal so my
application will be more tolerant of network delays and
less-than-ideal situations. In an attempt to understand latency
delays, I've boiled down my testcases to the bare minimium amount of
processing required.

I've used netcat and ffplay to remove as many factors as possible in
isolating sources of latency. Latency was measured by the viewer using
a stopwatch.

Baseline video test:

Streamer side:
  $ ./ffmpeg -f x11grab -s 1280x720 -i :0 -r 30 -deadline realtime
-b:v 1M -threads 7 -qmin 0 -qmax 50 -crf 5 -f ivf pipe:1 | nc *target
machine*

 Viewer side:
  $ nc -l *port* | ffplay -i ivf

After consider research, I found I could use the ivf container to get
the bare minimium required to record/stream VP8 data. What I've found
is the following

 1. Under best conditions, latency for recording from x11grab is about
500-600ms. Encoding time to VP8 is roughly 100ms.
 2. When recording from V4L2, or transcoding from media that is YUV
based, the latency drops to <100ms.
 3. x11grab uses the MIT-SHM extension to grab the raw RGB data from
X11. This has to be transcoded to YUV for almost any encoder to work
(x264rgb appears to be an exception).

 With a variety of encoder settings, the "base latency" is about
700ms, most of which I believe is caused by x11grab. Use of other
codecs as a test have shown similar results. Going further, I've
identified other points that have caused latency, including the use of
webm and such.

 Container latency test:
   $ ./ffmpeg -f x11grab -s 1280x720 -i :0 -r 30 -deadline realtime
-b:v 1M -threads 7 -qmin 0 -qmax 50 -crf 5 -f webm pipe:1 | nc *target
machine*

 Viewer side:
  $ nc -l *port* | ffplay -i webm

When switching to using webm with only a video stream, we found that
another 400ms of delay was added. Research suggests thats because webm
is a space-based container, and not really well suited to low latency
delays. This has brought our total latency delay to approximately
1-1.3 seconds.

Adding audio:

Ideally, we'd like to stream audio together with video in a container.
Unfortunately, this added a even worse delay. I tested multiple
codecs, and opus gave us the "best" results, though it was still not
great.

Audio latency test:
 $ ./ffmpeg -f x11grab -s 1280x720 -i :0 -f alsa -i pulse -acodec
libopus -deadline realtime -vcodec libvpx -deadline realtime -b:v 1M
-threads 7 -qmin 0 -qmax 50 -crf 5 -f webm  pipe:1

 Viewer side:
  $ nc -l *port* | ffplay -i webm

Adding opus audio brought our total latency up to approximately two
seconds, though this would vary. From the viewer side, audio and video
were synced and not dropping frames, it presented itself as a constant
delay.

Other codecs and containers were tried, getting roughly similar
results across the spectrum.

What I'd like to determine is the following:
 - Is it possible to get better performance from x11grab? As best I
can tell, most of the latency appears to becoming from having to
transcode from RGB to YUV formats internally within FFmpeg; grabbing a
raw YUV video source did not present the same latency delay
 - The muxer latency hit is painful. If we could get x11grab to go
faster, it would be livable, though adding audio has still caused us
to be far outside out latency target
 - Is there a better way to handle this situation? We could handle
doing two independent streams, one for audio, and one for video.

 FFmpeg on the streamer side was compiled as follows:

 ffmpeg version N-75472-g7086154 Copyright (c) 2000-2015 the FFmpeg developers
  built with gcc 4.9.2 (Ubuntu 4.9.2-0ubuntu1~14.04)
  configuration: --enable-gpl --disable-stripping --enable-avresample
--enable-avisynth --enable-fontconfig --enable-gnutls --enable-ladspa
--enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca
--enable-libcdio --enable-libfreetype --enable-libfribidi
--enable-libgsm --enable-libmodplug --enable-libmp3lame
--enable-libopenjpeg --enable-libopus --enable-libpulse
--enable-librtmp --enable-libschroedinger --enable-libspeex
--enable-libssh --enable-libtheora --enable-libtwolame
--enable-libvorbis --enable-libvpx --enable-libwavpack
--enable-libwebp --enable-opengl --enable-x11grab --enable-libxvid
--enable-libdc1394 --enable-frei0r --enable-libx264 --enable-libsoxr
--enable-openal --prefix=/opt/ffmpeg --enable-static
  libavutil      55.  2.100 / 55.  2.100
  libavcodec     57.  3.100 / 57.  3.100
  libavformat    57.  2.100 / 57.  2.100
  libavdevice    57.  0.100 / 57.  0.100
  libavfilter     6.  6.100 /  6.  6.100
  libavresample   3.  0.  0 /  3.  0.  0
  libswscale      4.  0.100 /  4.  0.100
  libswresample   2.  0.100 /  2.  0.100
  libpostproc    54.  0.100 / 54.  0.100

Viewer side was using a binary build of FFmpeg for Windows with VP8
but I don't think decoding speed was a huge problem in terms of
latency. Any insight anyone could provide would be a huge help. Please
Cc replies to me, as I'm not subscribed to the list (though I will
check the list archives in case a message gets sent there).

Thanks for your time in advance,
Michael Casadevall