[FFmpeg-devel] [PATCH v2] fftools/ffmpeg_dec: Don't keep sending frame to filters in flushing

[Anton Khirnov]

Quoting Fei Wang (2023-06-26 09:24:58)
> Filter may has a limited frame pool size. Do not always send frame to
> filters without reaping.
> 
> Fix the regression of commit 5fa00b38e6.
> 
> Example cmd:
> $ ffmpeg -threads 20 -init_hw_device vaapi=hw:/dev/dri/renderD128   \
> -hwaccel_output_format vaapi -hwaccel vaapi -i avc_1080P.mp4        \
> -vf 'scale_vaapi=w=720:h=480' -f null -
> 
> Signed-off-by: Fei Wang <fei.w.wang at intel.com>
> ---
> Change:
> 1. rebase to master.
> 
>  fftools/ffmpeg_dec.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/fftools/ffmpeg_dec.c b/fftools/ffmpeg_dec.c
> index 85bf8dc536..a7ae51fed2 100644
> --- a/fftools/ffmpeg_dec.c
> +++ b/fftools/ffmpeg_dec.c
> @@ -803,6 +803,12 @@ int dec_packet(InputStream *ist, const AVPacket *pkt, int no_eof)
>          av_frame_unref(d->frame);
>          if (ret < 0)
>              goto finish;
> +
> +        // During flushing, break out to reap filter once send a frame to filters to
> +        // avoid drain out filter's output frame pool. Especially for HW filters which
> +        // always have a limited frame pool size.
> +        if (!pkt)
> +            return 0;

This is wrong and will leave the decoder in an inconsistent state.

-- 
Anton Khirnov