[FFmpeg-devel] [FFmpeg-cvslog] libavutil/hwcontext_vaapi: Add a new nv12 format map to support vulkan frame

[Xiang, Haihao]

On Tue, 2021-12-28 at 03:57 +0000, Chen, Wenbin wrote:
> > On 10/12/2021 16:05, Wenbin Chen wrote:
> > > ffmpeg | branch: master | Wenbin Chen <wenbin.chen at intel.com> | Tue
> > 
> > Dec  7 17:05:50 2021 +0800| [f3c9847c2754b7a43eb721c95e356a53085c2491]
> > > committer: Lynne
> > > 
> > > libavutil/hwcontext_vaapi: Add a new nv12 format map to support vulkan
> > 
> > frame
> > > 
> > > Vulkan will map nv12 to R8 and GR88, so add this map to vaapi to support
> > > vulkan frame.
> > > 
> > > Signed-off-by: Wenbin Chen <wenbin.chen at intel.com>
> > > 
> > > > 
> > 
> > http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=f3c9847c2754b7a
> > 43eb721c95e356a53085c2491
> > > ---
> > > 
> > >   libavutil/hwcontext_vaapi.c | 1 +
> > >   1 file changed, 1 insertion(+)
> > > 
> > > diff --git a/libavutil/hwcontext_vaapi.c b/libavutil/hwcontext_vaapi.c
> > > index 75acc851d6..994b744e4d 100644
> > > --- a/libavutil/hwcontext_vaapi.c
> > > +++ b/libavutil/hwcontext_vaapi.c
> > > @@ -992,6 +992,7 @@ static const struct {
> > >   } vaapi_drm_format_map[] = {
> > >   #ifdef DRM_FORMAT_R8
> > >       DRM_MAP(NV12, 2, DRM_FORMAT_R8,  DRM_FORMAT_RG88),
> > > +    DRM_MAP(NV12, 2, DRM_FORMAT_R8,  DRM_FORMAT_GR88),
> > >   #endif
> > >       DRM_MAP(NV12, 1, DRM_FORMAT_NV12),
> > >   #if defined(VA_FOURCC_P010) && defined(DRM_FORMAT_R16)
> > 
> > This looks very shady.  Shouldn't one or the other of these be NV21, with
> > the
> > second plane VU rather than UV?
> > 
> > - Mark
> 
> I add this because I see vulkan map RG88 and GR88 to the same format.
> ```
> { DRM_FORMAT_GR88,     VK_FORMAT_R8G8_UNORM     },
> { DRM_FORMAT_RG88,     VK_FORMAT_R8G8_UNORM     },
> ```
> I thinks you are right. One of them should be NV21. I should switch the
> position
> Of GR88 and RG88 in this map table so that VK_FORMAT_R8G8_UNORM can be
> mapped to DRM_FORMAT_RG88 rather than DRM_FORMAT_GR88.

Changing the mapping will have other issues, e.g. another hw context supports
nv21. I think the root cause is nv12 and nv21 have the same VkFormat in 
vk_pixfmt_map[].

    { AV_PIX_FMT_NV12, { VK_FORMAT_R8_UNORM, VK_FORMAT_R8G8_UNORM }}, 
    { AV_PIX_FMT_NV21, { VK_FORMAT_R8_UNORM, VK_FORMAT_R8G8_UNORM }},

So the pixel format should be taken into account when mapping drm format and
vulkan format

Thanks
Haihao