[FFmpeg-devel] [PATCH] avutil/hwcontext_cuda: fix YUV420P cuda_get_buffer

[Timo Rothenpieler]

On 14.05.2018 23:29, Marton Balint wrote:
> Regression since ece068a771ac3f725e854c681ecbef08e792addc.
> 
> Signed-off-by: Marton Balint <cus at passwd.hu>
> ---
>  libavutil/hwcontext_cuda.c | 9 ++++++---
>  1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/libavutil/hwcontext_cuda.c b/libavutil/hwcontext_cuda.c
> index cb5d15c9d9..e16d0a2b4b 100644
> --- a/libavutil/hwcontext_cuda.c
> +++ b/libavutil/hwcontext_cuda.c
> @@ -159,10 +159,13 @@ static int cuda_get_buffer(AVHWFramesContext *ctx, AVFrame *frame)
>          return res;
>  
>      // YUV420P is a special case.
> -    // Nvenc expects the U/V planes in swapped order from how ffmpeg expects them.
> +    // Nvenc expects the U/V planes in swapped order from how ffmpeg expects them, also chroma is half-aligned
>      if (ctx->sw_format == AV_PIX_FMT_YUV420P) {
> -        FFSWAP(uint8_t*, frame->data[1], frame->data[2]);
> -        FFSWAP(int, frame->linesize[1], frame->linesize[2]);
> +        int aligned_width = FFALIGN(ctx->width, CUDA_FRAME_ALIGNMENT);
> +        frame->data[2]     = frame->data[0] + aligned_width * ctx->height;
> +        frame->data[1]     = frame->data[2] + aligned_width * ctx->height / 4;
> +        frame->linesize[1] = aligned_width / 2;
> +        frame->linesize[2] = aligned_width / 2;
>      }

Are you sure it isn't already that way? I tested specially YUV420P and
could not make out any issues.

And if this is really necessary, please don't use the old aligned_width
term, it's incredibly confusing and made the old code a mess.
linesize[0] should be identical to it if I'm not mistaken.

So something like

frame->linesize[1] = frame->linesize[2] = frame->linesize[0] / 2;
frame->data[2]     = frame->data[1];
frame->data[1]     = frame->data[2] + frame->linesize[2] * ctx->height / 2;