[FFmpeg-devel] [PATCH] libavfilter/vf_scale_cuda: fix src_pitch for 10bit videos

[Philip Langdale]

On Mon, 13 May 2019 11:18:09 +0000
Sergey Svechnikov <svechnikov66 at gmail.com> wrote:

> When scaling a 10bit video using scale_cuda filter (witch uses pixel
> format AV_PIX_FMT_P010LE), the output video gets distorted. I think
> it has something to do with the differences in processing between
> cuda_sdk and ffnvcodec with cuda_nvcc (the problem appears after this
> commit
> https://github.com/FFmpeg/FFmpeg/commit/2544c7ea67ca9521c5de36396bc9ac7058223742).
> To solve the problem we should not divide the input frame planes'
> linesizes by 2 and leave them as they are. More info, samples and
> reproduction steps are here
> https://github.com/Svechnikov/ffmpeg-scale-cuda-10bit-problem ---
> libavfilter/vf_scale_cuda.c | 4 ++-- 1 file changed, 2 insertions(+),
> 2 deletions(-)
> 
> diff --git a/libavfilter/vf_scale_cuda.c b/libavfilter/vf_scale_cuda.c
> index c97a802..7fc33ee 100644
> --- a/libavfilter/vf_scale_cuda.c
> +++ b/libavfilter/vf_scale_cuda.c
> @@ -423,11 +423,11 @@ static int scalecuda_resize(AVFilterContext
> *ctx, break;
>      case AV_PIX_FMT_P010LE:
>          call_resize_kernel(ctx, s->cu_func_ushort, 1,
> -                           in->data[0], in->width, in->height,
> in->linesize[0]/2,
> +                           in->data[0], in->width, in->height,
> in->linesize[0], out->data[0], out->width, out->height,
> out->linesize[0]/2, 2);
>          call_resize_kernel(ctx, s->cu_func_ushort2, 2,
> -                           in->data[1], in->width / 2, in->height /
> 2, in->linesize[1]/2,
> +                           in->data[1], in->width / 2, in->height /
> 2, in->linesize[1], out->data[0] + out->linesize[0] * ((out->height +
> 31) & ~0x1f), out->width / 2, out->height / 2, out->linesize[1] / 4,
> 2); break;

Thanks for reporting the problem. I took a look and identified the
precise mistake I made. I dropped the `pixel_size` scaling factor
when setting `pitchInBytes`. Here is the fix I intend to apply.

diff --git a/libavfilter/vf_scale_cuda.c b/libavfilter/vf_scale_cuda.c
index c97a802ddc..ecfd6a1c92 100644
--- a/libavfilter/vf_scale_cuda.c
+++ b/libavfilter/vf_scale_cuda.c
@@ -357,7 +357,7 @@ static int call_resize_kernel(AVFilterContext *ctx,
CUfunction func, int channel .res.pitch2D.numChannels = channels,
         .res.pitch2D.width = src_width,
         .res.pitch2D.height = src_height,
-        .res.pitch2D.pitchInBytes = src_pitch,
+        .res.pitch2D.pitchInBytes = src_pitch * pixel_size,
         .res.pitch2D.devPtr = (CUdeviceptr)src_dptr,
     };

--phil