[FFmpeg-devel] [PATCH] lavu/hwcontext_cuda: declare support for argb/abgr/rgba/bgra

[Philip Langdale]

On Sat, 10 Jun 2023 19:59:35 -0700
Philip Langdale <philipl at overt.org> wrote:

> These are physically compatible formats and are potentially useful.
> 
> Signed-off-by: Philip Langdale <philipl at overt.org>
> ---
>  libavutil/hwcontext_cuda.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/libavutil/hwcontext_cuda.c b/libavutil/hwcontext_cuda.c
> index 5ae7711c94..22eb9f5513 100644
> --- a/libavutil/hwcontext_cuda.c
> +++ b/libavutil/hwcontext_cuda.c
> @@ -45,6 +45,10 @@ static const enum AVPixelFormat
> supported_formats[] = { AV_PIX_FMT_YUV444P16,
>      AV_PIX_FMT_0RGB32,
>      AV_PIX_FMT_0BGR32,
> +    AV_PIX_FMT_ARGB,
> +    AV_PIX_FMT_ABGR,
> +    AV_PIX_FMT_RGBA,
> +    AV_PIX_FMT_BGRA,
>  #if CONFIG_VULKAN
>      AV_PIX_FMT_VULKAN,
>  #endif

After discussion with Timo, pushed as:

> @@ -45,6 +45,8 @@ static const enum AVPixelFormat supported_formats[]
> = { AV_PIX_FMT_YUV444P16,
>      AV_PIX_FMT_0RGB32,
>      AV_PIX_FMT_0BGR32,
> +    AV_PIX_FMT_RGB32,
> +    AV_PIX_FMT_BGR32,
>  #if CONFIG_VULKAN
>      AV_PIX_FMT_VULKAN,
>  #endif

to fix that nvenc accepts RGB32 and BGR32 but the hwcontext does not.

RGB32 and BGR32 map to RGBA and BGRA on little-endian, so it achieves
the main result I was originally looking for as well.

--phil