[FFmpeg-devel] [PATCH] lavc/vaapi_encode: fix segfault

[Xiang, Haihao]

On Ma, 2023-01-30 at 08:55 +0100, Anton Khirnov wrote:
> Quoting Xiang, Haihao (2023-01-30 06:23:21)
> > From: Haihao Xiang <haihao.xiang at intel.com>
> > 
> > This is a regression since commit fbdba9a1a69fe4df413d9e9df1b11db522946e75
> > 
> > input_image is freed in vaapi_encode_wait() however it is still used in
> > commit fbdba9a1a69fe4df413d9e9df1b11db522946e75
> > 
> > $ ffmpeg -vaapi_device /dev/dri/renderD128 -f lavfi -i testsrc -vf
> > "format=nv12,hwupload" -c:v h264_vaapi -f null -
> > 
> > Note input_image will be freed in vaapi_encode_free() now.
> > 
> > Cc: Anton Khirnov <anton at khirnov.net>
> > Signed-off-by: Haihao Xiang <haihao.xiang at intel.com>
> > ---
> >  libavcodec/vaapi_encode.c | 3 ---
> >  1 file changed, 3 deletions(-)
> 
> Thank you, looks good.
> 
> Could swear I tested this, but apparently not :/

There is another issue caused by commit fbdba91, some commands for transcode
doesn't work now.

E.g.

$ ffmpeg -y -init_hw_device vaapi=hw:/dev/dri/renderD128 -hwaccel_output_format
vaapi -hwaccel vaapi -i input.mpg -c:v h264_vaapi out.h264

...
[mpeg2video @ 0x55991e9c5540] get_buffer() failed
[mpeg2video @ 0x55991e9c5540] get_buffer() failed (-12 (nil))
Error while decoding stream #0:0: Operation not permitted
...

User will has to use -extra_hw_frames option to require more hardware frames.

E.g.

$ ffmpeg -y -init_hw_device vaapi=hw:/dev/dri/renderD128 -hwaccel_output_format
vaapi -hwaccel vaapi -extra_hw_frames 8 -i input.mpg -c:v h264_vaapi out.h264

Is this also taken as an regression ? 

Thanks
Haihao