[FFmpeg-devel] [PATCH 0/5] VAAPI support infrastructure, encoders

[Mark Thompson]

Hi all,

This patch series contains the following:

* Some VAAPI infrastructure in libavutil.  Initialisation code, along 
with routines to handle mapping and copying to/from VA surfaces. 
Doesn't really fit as a public API, but has to go there because it's 
used by the parts in both libavfilter and libavcodec.

* VAAPI decode hwaccel helper.  This allows use of the VAAPI hwaccel 
with ffmpeg itself, rather than only through third-party use of libavcodec.

* VAAPI H.264 encoder.  This is mostly a token effort to see things 
working - I moved on to the H.265 encoder once it worked at all.  The 
other patch offered on this list recently (the one that looks 
suspiciously like an unacknowleged copy of libva 
test/encode/h264encode.c) would certainly offer better quality for now, 
though it lacks the integration to use the video processing at the same 
time.

* VAAPI H.265 encoder.  This is a simple working encoder: it only 
supports IDR and P frames at fixed QP so far, so the output is not very 
good.  I am intending to work on this to make it better, see below.

* VAAPI conversion filter.  It exists to allow sensible transforms 
(scale, colour-convert from RGB) using hardware in front of the encoder, 
and is currently useless for anything else because it can only output 
VAAPI surfaces.  Adding more outputs would be straightforward, though 
not very useful without being able to feed it VAAPI surfaces as input 
from a decoder (see below).  Another improvement would be to support 
more available VPP filters - deinterlacing, other colour transforms, etc.

All of this code was written by me alone.  Some inspiration but no code 
was taken from libva examples and from libyami (mostly around what codec 
options I am allowed to set).

Tested with Intel Quick Sync gen 7 (Haswell/4500U), gen 8 
(Braswell/N3700) and gen 9 (Skylake/6300), all with libva 1.6.2 and 
Linux 4.3.3 (current Debian stretch/testing).


Problems:

* The bits in libavutil don't really make a very good public API.  Needs 
documentation if it does become one.  "AVVAAPI" is a terrible namespace 
prefix to write everywhere.  There should probably be a common way to 
select the default VAAPI connection (top-level option on ffmpeg itself?).

* The configure tests need to make a more intelligent decision of when 
to enable things.  (H.265 encode is not present before libva 1.6.0, say.)

* There may well be memory leaks.  I don't fully understand how the 
reference counting and auxiliary buffers in AVFrames work, so it would 
be very helpful if someone familiar with them could have a look at it 
(and suggest a better method which I can implement, if appropriate).

* Timestamps do something funny - I pass the pts values through in what 
I think is the correct way, but without '-vsync 0' it perpetually 
complains that they are somehow wrong ("Past duration ... too large"). 
Maybe some of the clocks aren't running at the same rate?  Someone 
familiar with this can probably instantly explain what I've done wrong.

* The colour conversion isn't quite working as I would hope - it seems 
to saturate some colours?  This is probably fixable by being clearer 
about the colour spaces involved.

* The encoders aren't very good.


Things I intend to work on:

* Clean this up to get it integrated.

* Improve the H.265 encoder: B frames and some sort of GOP 
configuration, along with better bitrate control.  Not sure what would 
be useful to do after that; suggestions welcome.

* Investigate the various copies to/from VAAPI surfaces.  It should be 
possible to improve this, at least in some cases.


Examples:

./ffmpeg -hwaccel vaapi -i in.mp4 -f null /dev/null

Decode input using the vaapi hwaccel, throw the output away.

./ffmpeg -vsync 0 -f x11grab -s 1920x1080 -i :0.0+0,0 -an -c:v 
vaapi_hevc -qp 18 -idr_interval 120 out.mp4

Capture the X11 screen and encode it to H.265 at low QP.  s/hevc/h264/ 
to get H.264 output instead, which will work on non-Skylake.  (Remove 
'-vsync 0' to see timestamp problem mentioned above.)

./ffmpeg -vsync 0 -f x11grab -s 1920x1080 -i :0.0+0,0 -an -vf vaapi_conv 
-c:v vaapi_hevc -qp 18 -idr_interval 120 out.mp4

Same as previous, but with the colour-conversion from RGB to NV12 now 
happening in hardware using the vaapi_conv filter (VAAPI surfaces are 
passed between the filter and the encoder).

./ffmpeg -vsync 0 -hwaccel vaapi -i in.mp4 -vf vaapi_conv=size=1280x720 
-an -c:v vaapi_hevc -qp 30 -idr_interval 120 out.mp4

Decode input using the vaapi hwaccel, scale to 1280x720 with vaapi_conv 
and encode with H.265.

This works, but unfortunately not entirely in hardware.  The hwaccel 
setup very much wants a normal output from the hwaccel helper, so you 
end up copying the surface into a separate buffer, then back into a 
different surface on the filter input.  I'm not sure how to fix this 
without hacking the decoder side horribly (it needs to select the VAAPI 
output format very early, before the filter is set up).


Other random notes:

H.265/HEVC naming: what should I be using here?  Existing practice isn't 
very clear: it seems to be mostly H.264 (ITU naming) and HEVC (ISO/MPEG 
naming), but with some instances of the other one in each case.  I 
mildly prefer H.265, but I've used hevc in all variable names to try to 
stay consistent with (some of the) existing practice.  It would be easy 
to change if there is consensus the other way.

A warning to anyone playing with the H.265 encoder on Skylake: the 
drivers are not robust to errors!  It is not difficult to hang your GPU 
if you fiddle with the low-level parameters.  It did always recover for 
me (the kernel driver spots the hang and resets everything after a few 
seconds), but that sounds like undefined behaviour so YMMV.  (Easy 
reproduction case if anyone feels like debugging some drivers: set 
log2_diff_max_min_luma_coding_block_size to 3 rather than 2.)


All comments welcome.

Thanks,

- Mark