[FFmpeg-devel] [PATCH] avutil/hwcontext_cuda: fix YUV420P cuda_get_buffer

[Marton Balint]

On Tue, 15 May 2018, Timo Rothenpieler wrote:

> On 14.05.2018 23:29, Marton Balint wrote:
>> Regression since ece068a771ac3f725e854c681ecbef08e792addc.
>> 
>> Signed-off-by: Marton Balint <cus at passwd.hu>
>> ---
>>  libavutil/hwcontext_cuda.c | 9 ++++++---
>>  1 file changed, 6 insertions(+), 3 deletions(-)
>> 
>> diff --git a/libavutil/hwcontext_cuda.c b/libavutil/hwcontext_cuda.c
>> index cb5d15c9d9..e16d0a2b4b 100644
>> --- a/libavutil/hwcontext_cuda.c
>> +++ b/libavutil/hwcontext_cuda.c
>> @@ -159,10 +159,13 @@ static int cuda_get_buffer(AVHWFramesContext *ctx, AVFrame *frame)
>>          return res;
>>
>>      // YUV420P is a special case.
>> -    // Nvenc expects the U/V planes in swapped order from how ffmpeg expects them.
>> +    // Nvenc expects the U/V planes in swapped order from how ffmpeg expects them, also chroma is half-aligned
>>      if (ctx->sw_format == AV_PIX_FMT_YUV420P) {
>> -        FFSWAP(uint8_t*, frame->data[1], frame->data[2]);
>> -        FFSWAP(int, frame->linesize[1], frame->linesize[2]);
>> +        int aligned_width = FFALIGN(ctx->width, CUDA_FRAME_ALIGNMENT);
>> +        frame->data[2]     = frame->data[0] + aligned_width * ctx->height;
>> +        frame->data[1]     = frame->data[2] + aligned_width * ctx->height / 4;
>> +        frame->linesize[1] = aligned_width / 2;
>> +        frame->linesize[2] = aligned_width / 2;
>>      }
>
> Are you sure it isn't already that way? I tested specially YUV420P and
> could not make out any issues.

For 4K content linesize[0] is 3840, but linesize[1] becomes 2048 instead 
of 1920.

>
> And if this is really necessary, please don't use the old aligned_width
> term, it's incredibly confusing and made the old code a mess.
> linesize[0] should be identical to it if I'm not mistaken.
>
> So something like
>
> frame->linesize[1] = frame->linesize[2] = frame->linesize[0] / 2;
> frame->data[2]     = frame->data[1];
> frame->data[1]     = frame->data[2] + frame->linesize[2] * ctx->height / 2;

Ok, will use this.

Regards,
Marton