[FFmpeg-devel] [PATCH] avcodec/pthread_frame: fix hw_device_ctx leak

[Anton Khirnov]

Quoting Thomas Guillem (2021-12-02 20:44:33)
> 
> On Thu, Dec 2, 2021, at 15:05, Andreas Rheinhardt wrote:
> > 1. hw_device_ctx is in the public AVCodecContext, not in the private
> > AVCodecInternal.
> > 2. ctx->internal is intended to be allocated for all the internal
> > AVCodecContexts; the check is only there to ensure that we don't crash
> > if an allocation error happens.
> > 3. libavcodec only ever modifies avctx->hw_device_ctx at one place: in
> > avcodec_close() where is unreferences the reference (if any). So if it
> > was set in avcodec_open2() and NULL in avcodec_close() (i.e. immediately
> > before avcodec_close()), it has been modified somehow and you need to
> > find out who did it.
> > 4. Some parts of the code use hw_device_ctx to derive references from
> > it. Maybe one of these references did not get freed?
> > (5. What do Valgrind/ASAN say? When was the reference that leaked created?)
> 
> OK, all your point are valid, I missed a precious comment from the avcodec.h API:
> "For both encoders and decoders this field should be set before            
>  avcodec_open2() is called and must not be written to thereafter."
> 
> Hence, the incomprehension.
> 
> So, to sum up: the reference that leaks is created from VLC, from the get_format() callback.
> Looking at pthread_frame.c, it seems that get_format() is called with a AVCodecContext owned by a PerThreadContext, so not the main AVCodecContext. That is why avcodec_close() is not closing it.
> 
> I just discovered that it's not officially supported then, but I really don't see any other way to plug VAAPI between VLC and ffmpeg.
> 
> I explain: VLC can't know the hw_device that will be used when it calls avcodec_open2(). VLC has a similar struct than hw_device, called dec_device. For VAAPI, this struct hold the va_dpy that is created when the VLC avcodec module first create the video output, from the get_format() callback.
> 
> I see 2 possibilities:
> 1/ Patch the documentation and allow to set a hw_device_ctx from get_format(), review all use case of hw_device_ctx to check if there is no inconsistency with this new change.
> 2/ Use any other way to pass the va_dpy from the get_format()? I didn't see any, but maybe I missed something obvious.

2. is the right answer - you're supposed to set hw_frames_ctx, possibly
using the avcodec_get_hw_frames_parameters() helper function.

-- 
Anton Khirnov