[FFmpeg-devel] [PATCH] https://trac.ffmpeg.org/ticket/7030 fix

[Mark Thompson]

On 04/07/18 05:07, Li, Zhong wrote:
>> -----Original Message-----
>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On Behalf
>> Of Hendrik Leppkes
>> Sent: Wednesday, July 4, 2018 2:15 AM
>> To: FFmpeg development discussions and patches
>> <ffmpeg-devel at ffmpeg.org>
>> Subject: Re: [FFmpeg-devel] [PATCH] https://trac.ffmpeg.org/ticket/7030 fix
>>
>> On Tue, Jul 3, 2018 at 10:55 AM Li, Zhong <zhong.li at intel.com> wrote:
>>>
>>>> -----Original Message-----
>>>> From: ffmpeg-devel [mailto:ffmpeg-devel-bounces at ffmpeg.org] On
>>>> Behalf Of Yuri Palich
>>>> Sent: Friday, March 2, 2018 7:55 PM
>>>> To: ffmpeg-devel <ffmpeg-devel at ffmpeg.org>
>>>> Subject: [FFmpeg-devel] [PATCH] https://trac.ffmpeg.org/ticket/7030
>>>> fix
>>>>
>>>> From 7056d06412214bd601afb4b423d20ab6100e6ac5 Mon Sep 17
>> 00:00:00
>>>> 2001
>>>> From: palich2000 <y.palich.t at gmail.com>
>>>> Date: Fri, 2 Mar 2018 11:48:22 +0000
>>>> Subject: [PATCH] https://trac.ffmpeg.org/ticket/7030 fix
>>>>
>>>> ---
>>>>  libavcodec/qsvenc_h264.c | 15 +++++++++++++++
>>>>  1 file changed, 15 insertions(+)
>>>>
>>>> diff --git a/libavcodec/qsvenc_h264.c b/libavcodec/qsvenc_h264.c
>>>> index
>>>> 09e4c0e..5f60675 100644
>>>> --- a/libavcodec/qsvenc_h264.c
>>>> +++ b/libavcodec/qsvenc_h264.c
>>>> @@ -34,6 +34,7 @@
>>>>  #include "qsv.h"
>>>>  #include "qsv_internal.h"
>>>>  #include "qsvenc.h"
>>>> +#include "hwaccel.h"
>>>>
>>>>  typedef struct QSVH264EncContext {
>>>>      AVClass *class;
>>>> @@ -169,6 +170,16 @@ static const AVCodecDefault qsv_enc_defaults[]
>> = {
>>>>      { NULL },
>>>>  };
>>>>
>>>> +#define HWACCEL_QSV(codec) \
>>>> +    HW_CONFIG_HWACCEL(1, 1, 1, QSV,          QSV,
>> ff_
>>>> ## codec ##
>>>> _qsv_hwaccel)
>>>> +
>>>> +const AVHWAccel ff_h264_qsv_hwaccel = {
>>>> +    .name                 = "h264_qsv",
>>>> +    .type                 = AVMEDIA_TYPE_VIDEO,
>>>> +    .id                   = AV_CODEC_ID_H264,
>>>> +    .pix_fmt              = AV_PIX_FMT_QSV
>>>> +};
>>>> +
>>>>  AVCodec ff_h264_qsv_encoder = {
>>>>      .name           = "h264_qsv",
>>>>      .long_name      = NULL_IF_CONFIG_SMALL("H.264 / AVC /
>> MPEG-4
>>>> AVC /
>>>> MPEG-4 part 10 (Intel Quick Sync Video acceleration)"), @@ -186,5
>>>> +197,9 @@ AVCodec ff_h264_qsv_encoder = {
>>>>      .priv_class     = &class,
>>>>      .defaults       = qsv_enc_defaults,
>>>>      .caps_internal  = FF_CODEC_CAP_INIT_CLEANUP,
>>>> +    .hw_configs            = (const AVCodecHWConfigInternal*[]) {
>>>> +                       HWACCEL_QSV(h264),
>>>> +    NULL
>>>> +    },
>>>>      .wrapper_name   = "qsv",
>>>>  };
>>>> --
>>>> 1.8.3.1
>>>
>>> LGTM. Verified this fix # ticket 7030.
>>
>> The fact that we removed the fake HWAccels for a reason still stands.
>> Please talk to Mark Thompson before commiting anything like this.
>>
>> - Hendrik
> 
> Sure. Won't push such a patch without clear commit message and also need to fix other qsv encoders broken issue. (this patch is only available for h264.)
> Waiting Mark's comment/confirmation.

So I think the problem behind this is that in some configurations the encoder and decoder somehow can't use the same device at the same time, and it isn't carried between them.  I think that's something to do with using Redhat/Centos and therefore an ancient (pre-2014?) Linux version which lacks support for rendernodes, but I never managed to confirm that.  These cases work fine for me on both Windows and non-ancient Linux, and they will also work if the device gets passed from the decoder to encoder via hardware frames.

The hack in the patch above is adding decoder metadata to the encoder which allows the -init_hw_device option to end up with the device automatically applied to both the encoder and the decoder.  However, we never actually defined that metadata for encoders so this is definitely not appropriate.  There was some intent to define it for encoders too (indeed, there is some vestigial code in ffmpeg_hw.c which this is hooking to get the effect it wants), but it turned out that the only interesting metadata anyone actually wanted was to know whether an encoder can hardware or not, and that was added as AV_CODEC_CAP_HARDWARE / AV_CODEC_CAP_HYBRID.  The HWConfig parts for encoders were then left undefined because noone actually wanted to look at that information at runtime.

If you want to fix this, I suggest one or both of:
* Discuss what the HWConfig information for encoders should contain and how you would like people to use it, precisely define and document it, then modify ffmpeg_hw.c suitably to pass devices automatically based on it.
* Add a new option of some kind to work like -hwaccel_device for decoders and -filter_hw_device for filters.

Another possibility would be to add a qsv-specific hack in ffmpeg.c to make this particular case work (with no change to the libraries).  Not sure exactly what that would entail, but maybe something like the strstr() hack that was present in 3.4 because we hadn't sorted out what codec metadata to provide at that point.  (I'd prefer to avoid this given the effort that went into removing all of the specific code from ffmpeg.c, but it is a possibility.)

Thanks,

- Mark


PS: When thinking about this, note that at the AMF and NVENC encoders can also make use of an AVCodecContext.hw_device_ctx on input, with multiple types accepted in both cases.