[FFmpeg-devel] [PATCH, 4/7] lavu/hwcontext_vaapi: add vaapi_format_map support for 0YUV/Y210/Y410

[Mark Thompson]

On 04/12/2019 14:44, Linjie Fu wrote:
> VA_RT_FORMAT describes the desired sampling format for surface.
> 
> When creating surface, VA_RT_FORMAT will be used firstly to choose
> the expected fourcc/media_format for the surface. And the fourcc
> will be revised by the value of VASurfaceAttribPixelFormat.
> 
> Add vaapi_format_map support for new pixel_format.
> This is fundamental for both VA-API and QSV.
> 
> Signed-off-by: Linjie Fu <linjie.fu at intel.com>
> ---
>  libavutil/hwcontext_vaapi.c | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c
> index cf11764..296234b 100644
> --- a/libavutil/hwcontext_vaapi.c
> +++ b/libavutil/hwcontext_vaapi.c
> @@ -116,6 +116,13 @@ static const VAAPIFormatDescriptor vaapi_format_map[] = {
>  #endif
>      MAP(UYVY, YUV422,  UYVY422, 0),
>      MAP(YUY2, YUV422,  YUYV422, 0),
> +#ifdef VA_FOURCC_Y210
> +    MAP(Y210, YUV422_10, Y210, 0),
> +#endif
> +    MAP(AYUV, YUV444,    0YUV, 0),
> +#ifdef VA_FOURCC_Y410
> +    MAP(Y410, YUV444_10, Y410, 0),
> +#endif
>      MAP(411P, YUV411,  YUV411P, 0),
>      MAP(422V, YUV422,  YUV440P, 0),
>      MAP(444P, YUV444,  YUV444P, 0),

(I've now bought an Ice Lake machine so that I can actually investigate what's going on here properly.)

The real problem is that the formats advertised by the driver for H.265/VP9 YUV 4:4:4 (AYUV and Y410) are incorrect.  There isn't any alpha support anywhere in the hardware (none of the multilayer stuff), so using formats with alpha in them is wrong and makes all of the attempts to use it extremely confusing.

To fix that we need to change the driver to use formats which actually reflect reality.  This is mostly straightforward, because the inner code in the driver mainly cares about layout rather than what is actually in each channel.

Some patches which kindof work (not particularly well tested):

libva: <http://ixia.jkqxz.net/~mrt/4b8a1bb668ce23702f993f9ae7a8f96c/0001-Add-fourcc-values-for-YUV-4-4-4-formats.patch>
iHD: <http://ixia.jkqxz.net/~mrt/4b8a1bb668ce23702f993f9ae7a8f96c/0001-Do-not-advertise-support-for-nonexistent-alpha-chann.patch>

I've called the non-alpha formats XYUV and XV30 to match libdrm (see <https://cgit.freedesktop.org/mesa/drm/tree/include/drm/drm_fourcc.h#n164>).   The 12-bit case also needs some correction - it's currently using Y216 and I think assuming that the user knows they need to ignore the alpha channel and the lower bits elsewhere, but we might need to have distinct formats to make it work properly outside the driver.

- Mark

PS:  It might help to split out the 4:2:2 case (which doesn't have these problems because the formats there don't have fake alpha channels or unknown depth) to go forward, and look separately at fixing the driver for the others.