[FFmpeg-devel] [PATCH] ffmpeg_qsv: use -hwaccel_device to specify a device for VAAPI backend

Sun Sep 6 21:01:16 EEST 2020

On 03/09/2020 05:59, mypopy at gmail.com wrote:
> On Thu, Sep 3, 2020 at 9:21 AM Xiang, Haihao <haihao.xiang at intel.com> wrote:
>> On Wed, 2020-09-02 at 14:04 +0000, Soft Works wrote:
>>>> -----Original Message-----
>>>> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
>>>> Rogozhkin, Dmitry V
>>>> Sent: Wednesday, September 2, 2020 3:54 PM
>>>> To: ffmpeg-devel at ffmpeg.org
>>>> Subject: Re: [FFmpeg-devel] [PATCH] ffmpeg_qsv: use -hwaccel_device to
>>>> specify a device for VAAPI backend
>>>>
>>>> On Wed, 2020-09-02 at 07:31 +0000, Soft Works wrote:
>>>>>> -----Original Message-----
>>>>>> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
>>>>>> Rogozhkin, Dmitry V
>>>>>> Sent: Wednesday, September 2, 2020 9:00 AM
>>>>>> To: ffmpeg-devel at ffmpeg.org
>>>>>> Subject: Re: [FFmpeg-devel] [PATCH] ffmpeg_qsv: use -hwaccel_device
>>>>>> to specify a device for VAAPI backend
>>>>>>
>>>>>> On Wed, 2020-09-02 at 04:28 +0000, Xiang, Haihao wrote:
>>>>>>> On Wed, 2020-09-02 at 04:20 +0000, Soft Works wrote:
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On
>>>>
>>>> Behalf
>>>>>>
>>>>>> Of
>>>>>>>>> Haihao Xiang
>>>>>>>>> Sent: Wednesday, September 2, 2020 6:01 AM
>>>>>>>>> To: ffmpeg-devel at ffmpeg.org
>>>>>>>>> Cc: Haihao Xiang <haihao.xiang at intel.com>
>>>>>>>>> Subject: [FFmpeg-devel] [PATCH] ffmpeg_qsv: use
>>>>>>>>> -hwaccel_device to specify a device for VAAPI backend
>>>>>>>>>
>>>>>>>>> According to the FFmpeg documentation, -hwaccel_device can be
>>>>>>>>> used to select a device for hardware acceleration
>>>>>>>>>
>>>>>>>>> Example:
>>>>>>>>> ffmpeg -hwaccel qsv -hwaccel_device /dev/dri/renderD129 -c:v
>>>>>>>>> h264_qsv \ -i
>>>>>>>>> in.h264 -c:v hevc_qsv out.h265
>>>>>>>>
>>>>>>>> No. The docs say about hwaccel_device
>>>>>>>>
>>>>>>>> It can either refer to an existing device created with
>>>>>>>> -init_hw_device by name,
>>>>>>>>
>>>>>>>> => /dev/dri/renderD129 is not an existing device name
>>>>>>>>
>>>>>>>> or it can create a new device as if ‘-init_hw_device’
>>>>>>>> type:hwaccel_device were called immediately before.
>>>>>>>>
>>>>>>>> => /dev/dri/renderD129 is not a valid string for init_hw_device
>>>>>>
>>>>>> Well, it seems that's where documentation does not reflect the
>>>>>> reality since (as pointed by Haihao below)
>>>>>> https://trac.ffmpeg.org/wiki/Hardware/VAAPI
>>>>>> is full of contradictory examples.
>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hence, the patch doesn't make sense.
>>>>>>>
>>>>>>> in https://trac.ffmpeg.org/wiki/Hardware/VAAPI,
>>>>>>>
>>>>>>> If only one stream is being used, -hwaccel_device can also accept
>>>>>>> a device path directly
>>>>>>>
>>>>>>> ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -i ...
>>>>>>>
>>>>>>> I think it can be applied to qsv too.
>>>>>>>
>>>>>>
>>>>>> I agree. Part of ffmpeg help:
>>>>>>      -hwaccel hwaccel name  use HW accelerated decoding
>>>>>>      -hwaccel_device devicename  select a device for HW acceleration
>>>>>>
>>>>>> To me specifying device via -hwaccel_device in the way it is being
>>>>>> suggest in commit is pretty intuitive and is better than explain
>>>>>> people that there is - qsv_device for that purpose. Niceness of
>>>>>> -hwaccel_device is that it is not tight to particular acceleration,
>>>>>> hence easier to work with for end users.
>>>>>
>>>>> In case of QSV, the usage of hwaccel_device is already defined:
>>>>>
>>>>> device selects a value in ‘MFX_IMPL_*’. Allowed values are:
>>>>> auto sw  hw  auto_any hw_any hw2 hw3 hw4
>>>>>
>>>>> I agree that it would be more intuitive, but you can't just give it a
>>>>> different meaning from one day to another and break functionality.
>>>>
>>>> Adding new values does not break functionality! No one  suggests to drop
>>>> existing set of values (sw, hw, etc.) - these are remaining. But basically
>>>> suggestion is to extend allowed set to accept drm device specification on
>>>> Linux, that's it. Honestly, I don't see problem here.
>>>> Some background story: hw,hw2,hw3 allows to select adapter for msdk
>>>> library on Windows, but they don't allow to do that on Linux. On Linux all
>>>> these hw options are basically the same for the reason that device is being
>>>> specified externally via SetHandle call. On Linux only sw/hw makes a
>>>> difference (other story that sw msdk library is not actually released
>>>> currently
>>>> on Linux). That's the origin of the request: we need a good way to specify
>>>> adapter on Linux.
>>>
>>> As mentioned, there are cases where both needs to be specified on Windows:
>>> hw, hw2, hw3,.. plus adapter number.
>>>
>>> 'qsv_device' is for specifying adapter number on Windows and DRI on Linux
>>> 'hwaccel_device' is for specifying hw, hw2, hw3 (everywhere)
>>
>> According to https://trac.ffmpeg.org/ticket/7649, -qsv_device was added to
>> workaround device selection on Linux.
>>
>> On Windows, we should not use another option to specify the adapter number, the
>> adapter is 1:1 mapped to the hw impl. I think we should fix it if the current
>> QSV depends on -qsv_device on Window.
>>
>> https://github.com/Intel-Media-SDK/MediaSDK/blob/master/samples/sample_decode/src/pipeline_decode.cpp#L245-L264
>>
>>>
>>> I see no reason to start mixing that and allowing to specify DRI via
>>> 'hwaccel_device' parameter.
>>>
>>> That would make things just worse instead of improving anything.
>>
>>  From user's perspective, we may use the same device selection for both QSV and
>> VAAPI on Linux. It is easy for user to move from QSV to VAAPI or inverse.
>>
> Which case want to mix the  QSV and VAAPI in one FFmpeg pipeline? Do
> you have a real case for this?
> 
> I know we used  VAAPI as decoder and QSV as encoder before when QSV
> decoder have some limitation in FFmpeg
> 
> But I think we are better to fix QSV decoder in FFmpeg part or MSDK,
> do you have any other case to mix VAAPI and QSV?

Benefits of internal accelerated decoder:
- No need to find determine specific codec and support separately.
- Automatic software fallback for incomplete hardware implementations (like H.264 10-bit).
- Common metadata handling (HDR, 3D, closed captions, timecodes, ...).
- Decode-only setup does not change based on the device manufacturer.

Benefits of external offload decoder:
- Slightly easier command lines for fixed simple cases.
- Per-device ad-hoc tuning can be applied elsewhere, in a way that would be hard inside libavcodec (e.g. of decoder output delay for maximising single-stream throughput).

IMO the first set of benefits are more valuable to most users.

- Mark