[FFmpeg-devel] [PATCH v2 3/5] avutil/hwcontext_d3d11va: add a format check for staging texture

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of Tong
> Wu
> Sent: Friday, April 29, 2022 12:45 PM
> To: ffmpeg-devel at ffmpeg.org
> Cc: Tong Wu <tong1.wu at intel.com>
> Subject: [FFmpeg-devel] [PATCH v2 3/5] avutil/hwcontext_d3d11va: add a
> format check for staging texture
> 
> The texDesc.Format needs to be filled in with a corresponding format.
> I
> add a format check to initialize the format in case sometimes the
> ctx->internal->priv is not initialized, such as during the hwmap
> process.

ctx->internal->priv is D3D11VAFramesContext. When it wouldn't be 
initialized, then hwmap couldn't work.

D3D11VAFramesContext.format is set during d3d11va_frames_init.
You would need to find out whether init is not called or
whether AVHWFramesContext.sw_format is not (yet) set during 
init.


If that doesn't work out for some reason, I think the next best
solution would be to add a 'format parameter' to 
d3d11va_create_staging_texture() and in d3d11va_transfer_data()
(the only caller) do ID3D11Texture2D_GetDesc() on the frame 
texture ('texture' variable) and pass the returned format to
d3d11va_create_staging_texture()

Kind regards,
softworkz


> 
> $ ffmpeg.exe -y -hwaccel qsv -init_hw_device d3d11va=d3d11 \
> -init_hw_device qsv=qsv at d3d11 -c:v h264_qsv \
> -i input.h264 -vf
> "hwmap=derive_device=d3d11va,format=d3d11,hwdownload,format=nv12" \
> -f null -
> 
> Signed-off-by: Tong Wu <tong1.wu at intel.com>
> ---
>  libavutil/hwcontext_d3d11va.c | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/libavutil/hwcontext_d3d11va.c
> b/libavutil/hwcontext_d3d11va.c
> index db529acbb4..0ec0e07d3a 100644
> --- a/libavutil/hwcontext_d3d11va.c
> +++ b/libavutil/hwcontext_d3d11va.c
> @@ -349,6 +349,8 @@ static int
> d3d11va_create_staging_texture(AVHWFramesContext *ctx)
>      AVD3D11VADeviceContext *device_hwctx = ctx->device_ctx->hwctx;
>      D3D11VAFramesContext              *s = ctx->internal->priv;
>      HRESULT hr;
> +    int i;
> +
>      D3D11_TEXTURE2D_DESC texDesc = {
>          .Width          = ctx->width,
>          .Height         = ctx->height,
> @@ -360,6 +362,20 @@ static int
> d3d11va_create_staging_texture(AVHWFramesContext *ctx)
>          .CPUAccessFlags = D3D11_CPU_ACCESS_READ |
> D3D11_CPU_ACCESS_WRITE,
>      };
> 
> +    if (!texDesc.Format) {
> +        for (i = 0; i < FF_ARRAY_ELEMS(supported_formats); i++) {
> +            if (ctx->sw_format == supported_formats[i].pix_fmt) {
> +                texDesc.Format = supported_formats[i].d3d_format;
> +                break;
> +            }
> +        }
> +        if (i == FF_ARRAY_ELEMS(supported_formats)) {
> +            av_log(ctx, AV_LOG_ERROR, "Unsupported pixel format:
> %s\n",
> +                av_get_pix_fmt_name(ctx->sw_format));
> +            return AVERROR(EINVAL);
> +        }
> +    }
> +
>      hr = ID3D11Device_CreateTexture2D(device_hwctx->device, &texDesc,
> NULL, &s->staging_texture);
>      if (FAILED(hr)) {
>          av_log(ctx, AV_LOG_ERROR, "Could not create the staging
> texture (%lx)\n", (long)hr);