[FFmpeg-devel] [PATCH] avcodec/pthread_frame: fix hw_device_ctx leak

[Thomas Guillem]

On Thu, Dec 2, 2021, at 15:05, Andreas Rheinhardt wrote:
> 1. hw_device_ctx is in the public AVCodecContext, not in the private
> AVCodecInternal.
> 2. ctx->internal is intended to be allocated for all the internal
> AVCodecContexts; the check is only there to ensure that we don't crash
> if an allocation error happens.
> 3. libavcodec only ever modifies avctx->hw_device_ctx at one place: in
> avcodec_close() where is unreferences the reference (if any). So if it
> was set in avcodec_open2() and NULL in avcodec_close() (i.e. immediately
> before avcodec_close()), it has been modified somehow and you need to
> find out who did it.
> 4. Some parts of the code use hw_device_ctx to derive references from
> it. Maybe one of these references did not get freed?
> (5. What do Valgrind/ASAN say? When was the reference that leaked created?)

OK, all your point are valid, I missed a precious comment from the avcodec.h API:
"For both encoders and decoders this field should be set before            
 avcodec_open2() is called and must not be written to thereafter."

Hence, the incomprehension.

So, to sum up: the reference that leaks is created from VLC, from the get_format() callback.
Looking at pthread_frame.c, it seems that get_format() is called with a AVCodecContext owned by a PerThreadContext, so not the main AVCodecContext. That is why avcodec_close() is not closing it.

I just discovered that it's not officially supported then, but I really don't see any other way to plug VAAPI between VLC and ffmpeg.

I explain: VLC can't know the hw_device that will be used when it calls avcodec_open2(). VLC has a similar struct than hw_device, called dec_device. For VAAPI, this struct hold the va_dpy that is created when the VLC avcodec module first create the video output, from the get_format() callback.

I see 2 possibilities:
1/ Patch the documentation and allow to set a hw_device_ctx from get_format(), review all use case of hw_device_ctx to check if there is no inconsistency with this new change.
2/ Use any other way to pass the va_dpy from the get_format()? I didn't see any, but maybe I missed something obvious.

Best,
Thomas

>
> - Andreas
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".