[FFmpeg-devel] [PATCH 2/2] avcodec/pthread_frame: Only attempt to close decoders which have allocated private data

[Andreas Rheinhardt]

Michael Niedermayer:
> Fixes: Null pointer dereference
> Fixes: ff_h264_remove_all_refs.mp4
> 
> Found-by: Rafael Dutra <rafael.dutra at cispa.de>
> Tested-by: Rafael Dutra <rafael.dutra at cispa.de>
> Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> ---
>  libavcodec/pthread_frame.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/libavcodec/pthread_frame.c b/libavcodec/pthread_frame.c
> index 7bcb9a7bcc..048e535cb6 100644
> --- a/libavcodec/pthread_frame.c
> +++ b/libavcodec/pthread_frame.c
> @@ -708,7 +708,7 @@ void ff_frame_thread_free(AVCodecContext *avctx, int thread_count)
>              pthread_join(p->thread, NULL);
>          p->thread_init=0;
>  
> -        if (codec->close && p->avctx)
> +        if (codec->close && p->avctx && p->avctx->priv_data)
>              codec->close(p->avctx);
>  
>  #if FF_API_THREAD_SAFE_CALLBACKS
> 
This does not fix the whole issue: A codec without
FF_CODEC_CAP_INIT_CLEANUP set might already have cleaned up internally
on error and it might not be safe to do it again; and calling close
before having called init (if existing) is not allowed for any codec.

I have already sent a patch for this (which needs to be slightly updated
due to James having added a new failure path (an av_packet_alloc)) here:
https://patchwork.ffmpeg.org/project/ffmpeg/patch/20210211155759.309391-1-andreas.rheinhardt@gmail.com/
I will see whether Nuo's suggestions would lead to an improvement.

- Andreas