[FFmpeg-devel] [PATCH] hwcontext_vaapi: introduce AV_HWFRAME_MAP_DRM_COMPOSED_LAYERS

[Simon Ser]

This allows callers to ask for composed layers. From the libva
docs:

> If VA_EXPORT_SURFACE_SEPARATE_LAYERS is specified on export, each
> layer will contain exactly one plane.  For example, an NV12
> surface will be exported as two layers, one of DRM_FORMAT_R8 and
> one of DRM_FORMAT_GR88.
> If VA_EXPORT_SURFACE_COMPOSED_LAYERS is specified on export,
> there will be exactly one layer.

VA_EXPORT_SURFACE_COMPOSED_LAYERS is desirable in many scenarios,
for instance when directly importing the DMA-BUFs into APIs such
as GL (as a single EGLImage), KMS or Wayland.

This patch was tested with the following Mesa MR:
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/9015

Signed-off-by: Simon Ser <contact at emersion.fr>
Cc: Mark Thompson <sw at jkqxz.net>
Cc: Haihao Xiang <haihao.xiang at intel.com>
Cc: Fei Wang <fei.w.wang at intel.com>
---
 libavutil/hwcontext.h       | 6 ++++++
 libavutil/hwcontext_vaapi.c | 6 +++++-
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/libavutil/hwcontext.h b/libavutil/hwcontext.h
index 04d19d89c2b8..9ea27b285c20 100644
--- a/libavutil/hwcontext.h
+++ b/libavutil/hwcontext.h
@@ -538,6 +538,12 @@ enum {
      * be much lower than normal memory.
      */
     AV_HWFRAME_MAP_DIRECT    = 1 << 3,
+    /**
+     * The mapping must result in exactly one DRM layer, see
+     * AVDRMFrameDescriptor.  For instance, an NV12 frame will have a single
+     * layer whose format is NV12.
+     */
+    AV_HWFRAME_MAP_DRM_COMPOSED_LAYERS = 1 << 4,
 };
 
 /**
diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c
index 2227d6ed6981..0d94dc9f14de 100644
--- a/libavutil/hwcontext_vaapi.c
+++ b/libavutil/hwcontext_vaapi.c
@@ -1161,7 +1161,11 @@ static int vaapi_map_to_drm_esh(AVHWFramesContext *hwfc, AVFrame *dst,
 
     surface_id = (VASurfaceID)(uintptr_t)src->data[3];
 
-    export_flags = VA_EXPORT_SURFACE_SEPARATE_LAYERS;
+    export_flags = 0;
+    if (flags & AV_HWFRAME_MAP_DRM_COMPOSED_LAYERS)
+        export_flags |= VA_EXPORT_SURFACE_COMPOSED_LAYERS;
+    else
+        export_flags |= VA_EXPORT_SURFACE_SEPARATE_LAYERS;
     if (flags & AV_HWFRAME_MAP_READ)
         export_flags |= VA_EXPORT_SURFACE_READ_ONLY;
     if (flags & AV_HWFRAME_MAP_WRITE)
-- 
2.30.0