[FFmpeg-devel] [PATCH] avcodec/nvenc: fix NULL pointer dereference

[Zhao Zhili]

> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of Timo Rothenpieler
> Sent: 2023年2月28日 22:37
> To: ffmpeg-devel at ffmpeg.org
> Subject: Re: [FFmpeg-devel] [PATCH] avcodec/nvenc: fix NULL pointer dereference
> 
> On 28.02.2023 19:43, Zhao Zhili wrote:
> > From: Zhao Zhili <zhilizhao at tencent.com>
> >
> > This can happen if encoder init failed before setup the queue, then
> > reorder_queue_flush() is called by close().
> >
> > Fix ticket #10221
> >
> > Signed-off-by: Zhao Zhili <zhilizhao at tencent.com>
> > ---
> >   libavcodec/nvenc.c | 2 ++
> >   1 file changed, 2 insertions(+)
> >
> > diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
> > index 8a28454042..dda2dc3ba1 100644
> > --- a/libavcodec/nvenc.c
> > +++ b/libavcodec/nvenc.c
> > @@ -178,6 +178,8 @@ static void reorder_queue_flush(AVFifo *queue)
> >   {
> >       FrameData fd;
> >
> > +    if (!queue)
> > +        return;
> >       while (av_fifo_read(queue, &fd, 1) >= 0)
> >           av_buffer_unref(&fd.frame_opaque_ref);
> >   }
> 
> This should probably be an assert0, and the function should not be
> called to begin with if the queue is NULL.

I can move the check from here to close(). assert is helpful if it can catch
bugs during development. It doesn't help much here since this is a rare case 
which doesn't triggered during development.

> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".