[FFmpeg-devel] [PATCH] DXVA2: Fix crash releasing IDirect3D9Surface's with av_buffer_default_free instead of Release

[Hendrik Leppkes]

Please don't top post on the mailing list.

On Tue, Oct 4, 2016 at 8:20 AM, Min <yosoymin at gmail.com> wrote:
> Hi, I'm using ffmpeg dxva2 implementation (ffmpeg_dxva2.c) to render mp4
> video to a DX texture.
>
> The crash happens when closing my application, when I call to
> av_frame_free() to the last frame I've created to decompress video. The
> hwcontext references go to 0 and internal structs start to free its memory,
> but with DXVA2FramesContext::surfaces_internal I've noticed that is an
> array with IDirect3D9Surfaces's. You can see it in the file
> libavutil/hwcontext_dxva2.c:169, inside function dxva2_init_pool().
>
>     hr = IDirectXVideoAccelerationService_CreateSurface(s->service,
>                                                         ctx->width,
> ctx->height,
>
> ctx->initial_pool_size - 1,
>                                                         s->format,
> D3DPOOL_DEFAULT, 0,
>
> frames_hwctx->surface_type,
>
> s->surfaces_internal, NULL);
>
> In the same file, the function dxva2_pool_alloc() calls av_buffer_create()
> with every surface created before, but the "free" function pointer
> parameter is passed as NULL, and in that case av_buffer_default_free() will
> be called for every surface, and that's not correct. I've created the patch
> to avoid this case, and I've checked that there is no memory leaks using
> debug CRT with visual studio. The surfaces are properly released in the
> function dxva2_frames_uninit().

The surfaces are always released, this is about memory, not surfaces.
It sounds to me like your copy of the implementation here is just
flawed, and you need to increase the refcount of the pool somewhere.

The hwcontext implementation seems fine. The free is required or it'll
leak memory.

>
> I don't know exactly how to reproduce using ffmpeg command line because
> what I'm doing is integrating ffmeg in my 3d engine, but I supose that if
> you decompress using dxva2 it will crash when finishing ffmpeg app, if all
> the reference counting is ok and all the context and frames are freed.
>

the ffmpeg app does not crash when you use dxva2 decoding.

- Hendrik