[FFmpeg-devel] ffmpeg-qsv: Is opaque memory allocation path used in ffmpeg-qsv?

[Rogozhkin, Dmitry V]

On Fri, 2019-10-25 at 02:17 +0000, Fu, Linjie wrote:
> > -----Original Message-----
> > From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> > Rogozhkin, Dmitry V
> > Sent: Friday, October 25, 2019 05:17
> > To: ffmpeg-devel at ffmpeg.org
> > Subject: Re: [FFmpeg-devel] ffmpeg-qsv: Is opaque memory allocation
> > path
> > used in ffmpeg-qsv?
> > 
> > On Thu, 2019-10-24 at 17:12 +0200, Max Dmitrichenko wrote:
> > > Hi,
> > > 
> > > Just have a look at any usage as: ffmpeg -hwaccel qsv .......
> > 
> > I looked into these 3 command lines on Linux:
> > 
> > ffmpeg -hwaccel qsv -c:v h264_qsv -i AUD_MW_E.264  \
> >        -vf hwdownload,format=nv12 -pix_fmt yuv420p \
> >        AUD_MW.yuv
> > 
> > ffmpeg -hwaccel qsv -c:v h264_qsv -i AUD_MW_E.264 \
> >        -c:v hevc_qsv -b:v 5M AUD_MW_E.hevc
> > 
> > ffmpeg -init_hw_device qsv=hw     \
> >        -filter_hw_device hw -f rawvideo -pix_fmt  \
> >        yuv420p -s:v 176x144 -i AUD_MW.yuv -vf     \
> >        hwupload=extra_hw_frames=64,format=qsv     \
> >        -c:v h264_qsv -b:v 5M -frames:v 10         \
> >        -y ./AUD_MW_E.h264
> > 
> > For what I see, none of the them is using QSV opaque memory. For
> > the
> > first 2 command lines ffmpeg explicitly prints (w/ -loglevel
> > debug):
> > 
> > [h264_qsv @ 0x34f6980] Decoder: output is video memory surface
> > 
> > Besides, I checked w/ manual printfs from the code - opaque is not
> > used.
> > 
> > Hence, this question: is actually qsv opaque memory path used
> > anywhere
> > in ffmpeg.exe sample or in other libavcodec/libavfilter based
> > applications? Is there Linux/Windows difference which I don't
> > notice in
> > code review?
> > 
> 
> I didn't see MFX_MEMTYPE_OPAQUE_FRAME set for frames_hwctx-
> >frame_type
> in mainline ffmpeg application.

Right. I also don't see it. I think that none of upstream ffmpeg
applications actually exercise this path. This raises a concern why we
have this in sources and complicate them if we don't use them? I guess
most testing is done via mainline ffmpeg.exe and opaque path test
coverage is likely missing meaning it might not be actually functional.

If we will look from the end-user perspective, there is also
opportunity to write their own applications using libavcodec/etc.
libraries. In this case he will work with these 2 header files exposing
QSV capabilities:

./libavcodec/qsv.h
./libavutil/hwcontext_qsv.h

On this path he will be able to use the following:
AVQSVFramesContext::frame_type
AVQSVContext::opaque_alloc
to configure QSV.

Questions:
1. Is opaque memory usage limited to internal ffmpeg component
connections? or it is possible to have external non-ffmpeg component
connected to ffmpeg one via qsv opaque memory?
2. Does anyone actually uses the above to configure qsv opaque memory?
3. Regardless of the above: is that possible to deprecate and drop QSV
opaque memory support in ffmpeg-qsv path?

> 
> BTW, in the frame_allocator used for
> MFXVideoCORE_SetFrameAllocator(),
> https://github.com/FFmpeg/FFmpeg/blob/6f2a3958cfac135c60b509a61a4fd39432d8f9a9/libavcodec/qsv.c#L562
> 
> only MFX_MEMTYPE_INTERNAL_FRAME and MFX_MEMTYPE_EXTERNAL_FRAME is
> supported as an allocation request since commit:
> 4ab61cd983b539749bd621ea271624ddb5196a8e
> 
> (For VPP,  MFX_MEMTYPE_FROM_VPPIN is supported)
> - linjie
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".