[FFmpeg-devel] [PATCH] lavf/vf_deinterlace_vaapi: flush queued frame in field mode

[Fu, Linjie]

> -----Original Message-----
> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On Behalf
> Of Mark Thompson
> Sent: Wednesday, August 21, 2019 04:49
> To: ffmpeg-devel at ffmpeg.org
> Subject: Re: [FFmpeg-devel] [PATCH] lavf/vf_deinterlace_vaapi: flush
> queued frame in field mode
> 
> On 02/08/2019 10:53, Linjie Fu wrote:
> > Add deint_vaapi_request_frame for deinterlace_vaapi, send NULL frame
> > to flush the queued frame.
> >
> > Fix the frame drop issue in field mode:
> >
> > ffmpeg -hwaccel vaapi -vaapi_device /dev/dri/renderD128 -v verbose -c:v
> > h264 -i ./input.h264 -vf 'format=nv12|vaapi,hwupload,
> >         deinterlace_vaapi=mode=bob:rate=field,hwdownload,format=nv12'
> > -pix_fmt yuv420p -f rawvideo -vsync passthrough -y dump.yuv
> >
> > Signed-off-by: Linjie Fu <linjie.fu at intel.com>
> > ---
> >  libavfilter/vf_deinterlace_vaapi.c | 46
> ++++++++++++++++++++++++++++++++------
> >  1 file changed, 39 insertions(+), 7 deletions(-)
> 
> Can you explain in more detail what the problem is here?  What frame is
> being dropped?

For DeinterlacingBob mode with rate=field, the frame number of output
should equal 2x input total since only intra deinterlace is used.

Currently for "pipeline_caps.num_backward_references, rate = field",
extra_delay is introduced. Due to the async without flush, frame number
of output is [expected_number - 2].

Specifically, if the input only has 1 frame, the output will be empty.

For 1 frame input in Bob mode with rate=field,
before patch: 0 frame;
after  patch: 2 frames;

> 
> I already get the expected number of frames out with both rate settings
> (input total - forward references - backward references, all multiplied by 2
> for field rate output).  E.g. for 100 frames/field-pairs of input:
> 
> $ ./ffmpeg_g -hwaccel vaapi -hwaccel_device /dev/dri/renderD129 -
> hwaccel_output_format vaapi -i f100.mp4 -an -vf
> deinterlace_vaapi=rate=frame -f null -
> ...
> frame=   97
> 
> $ ./ffmpeg_g -hwaccel vaapi -hwaccel_device /dev/dri/renderD129 -
> hwaccel_output_format vaapi -i f100.mp4 -an -vf
> deinterlace_vaapi=rate=field -f null -
> ...
> frame=  194
> 
> (With forward = 2, backward = 1.)

Which driver is used for this test?

I've verified with i965 and iHD on linux and failed to find the matched
pipeline_cap with num_forward_reference = 2, num_backward_reference = 1;

For iHD:

#define DDI_CODEC_NUM_FWD_REF 0;
#define DDI_CODEC_NUM_BK_REF 0;

num_forward_reference = DDI_CODEC_NUM_FWD_REF; 
num_backward_reference = DDI_CODEC_NUM_BK_REF;

https://github.com/intel/media-driver/blob/b9606584bf3dc6937047b4f19e08c75f88bd95a4/media_driver/linux/common/ddi/media_libva.cpp#L5687

For i965:
num_forward_reference ++; (if working on MotionAdaptive or MotionCompensated)
num_backward_reference = 0;

https://github.com/intel/intel-vaapi-driver/blob/9bc30a0231e55f17afed50589669d11e844d0bb9/src/i965_drv_video.c#L7158

> 
> With this patch applied, the filter always segfaults for me at the end of the
> stream when set to field rate:
> 
> $ gdb --args ./ffmpeg_g -hwaccel vaapi -hwaccel_device
> /dev/dri/renderD129 -hwaccel_output_format vaapi -i f100.mp4 -an -vf
> deinterlace_vaapi=rate=field -f null -
> ...
> Thread 1 "ffmpeg_g" received signal SIGSEGV, Segmentation fault.
> 0x0000555555755fb1 in deint_vaapi_filter_frame (inlink=0x555559092300,
> input_frame=0x555558d42000) at src/libavfilter/vf_deinterlace_vaapi.c:227
> 227                 ctx->frame_queue[current_frame_index + i + 1]->data[3];
> (gdb) bt
> #0  0x0000555555755fb1 in deint_vaapi_filter_frame (inlink=0x555559092300,
> input_frame=0x555558d42000) at src/libavfilter/vf_deinterlace_vaapi.c:227
> #1  0x00005555557565d2 in deint_vaapi_request_frame
> (link=0x5555590928c0) at src/libavfilter/vf_deinterlace_vaapi.c:342
> #2  0x00005555556ce950 in ff_request_frame_to_filter (link=0x5555590928c0)
> at src/libavfilter/avfilter.c:458
> #3  0x00005555556d0cd0 in forward_status_change (filter=0x555559088940,
> in=0x555559092300) at src/libavfilter/avfilter.c:1243
> #4  0x00005555556d0eaf in ff_filter_activate_default (filter=0x555559088940)
> at src/libavfilter/avfilter.c:1274
> #5  0x00005555556d0fec in ff_filter_activate (filter=0x555559088940) at
> src/libavfilter/avfilter.c:1430
> #6  0x00005555556d5d29 in ff_filter_graph_run_once (graph=0x55555908ff00)
> at src/libavfilter/avfiltergraph.c:1456
> #7  0x00005555556d71d8 in push_frame (graph=0x55555908ff00) at
> src/libavfilter/buffersrc.c:187
> #8  0x00005555556d77a9 in av_buffersrc_close (ctx=0x555559091ac0,
> pts=300300, flags=4) at src/libavfilter/buffersrc.c:275
> #9  0x000055555569356a in ifilter_send_eof (ifilter=0x5555580d0e40,
> pts=300300) at src/fftools/ffmpeg.c:2213
> #10 0x00005555556949fd in send_filter_eof (ist=0x5555580f3ac0) at
> src/fftools/ffmpeg.c:2562
> #11 0x0000555555695336 in process_input_packet (ist=0x5555580f3ac0,
> pkt=0x0, no_eof=0) at src/fftools/ffmpeg.c:2701
> #12 0x000055555569a9db in process_input (file_index=0) at
> src/fftools/ffmpeg.c:4313
> #13 0x000055555569c5f9 in transcode_step () at src/fftools/ffmpeg.c:4638
> #14 0x000055555569c726 in transcode () at src/fftools/ffmpeg.c:4692
> #15 0x000055555569cfb6 in main (argc=15, argv=0x7fffffffe488) at
> src/fftools/ffmpeg.c:4894
> (gdb) p current_frame_index
> $1 = 2
> (gdb) p i
> $2 = 0
> (gdb) p ctx->frame_queue[current_frame_index + i + 1]
> $3 = (AVFrame *) 0x0
> 

The last frame required a backward_reference which is not existed.
It seems that the flush is redundant under this condition
(pipeline_caps.num_backward_reference  > 0).

A possible solution is to flush the queue only when extra_delay is needed
for timestamps.(Not verified yet since couldn't got the same pipeline_caps 
environment )

New patch will be sent later.

- linjie