[FFmpeg-devel] [PATCH] hwcontext_videotoolbox: add vt_device_derive

[Mark Thompson]

On 22/02/2024 19:39, ChenLiucheng via ffmpeg-devel wrote:
>> On Feb 23, 2024, at 03:28, Mark Thompson <sw at jkqxz.net> wrote:
>>
>> On 22/02/2024 18:46, gnattu via ffmpeg-devel wrote:
>>> There is no device context to be setup, nor devices to be
>>> selected with VideoToolbox. Just a simple return would allow
>>> us to use derived device in filters like
>>> `hwupload=derive_device=videotoolbox`
>>> Signed-off-by: Gnattu OC <gnattuoc at me.com>
>>> ---
>>>   libavutil/hwcontext_videotoolbox.c | 9 +++++++++
>>>   1 file changed, 9 insertions(+)
>>> diff --git a/libavutil/hwcontext_videotoolbox.c b/libavutil/hwcontext_videotoolbox.c
>>> index fe469dc161..d13199eca7 100644
>>> --- a/libavutil/hwcontext_videotoolbox.c
>>> +++ b/libavutil/hwcontext_videotoolbox.c
>>> @@ -759,6 +759,14 @@ static int vt_device_create(AVHWDeviceContext *ctx, const char *device,
>>>       return 0;
>>>   }
>>>   +static int vt_device_derive(AVHWDeviceContext *device_ctx,
>>> +                            AVHWDeviceContext *src_ctx, AVDictionary *opts,
>>> +                            int flags)
>>> +{
>>> +    // There is no context to be setup with VT, just return.
>>> +    return 0;
>>> +}
>>> +
>>>   const HWContextType ff_hwcontext_type_videotoolbox = {
>>>       .type                 = AV_HWDEVICE_TYPE_VIDEOTOOLBOX,
>>>       .name                 = "videotoolbox",
>>> @@ -766,6 +774,7 @@ const HWContextType ff_hwcontext_type_videotoolbox = {
>>>       .frames_priv_size     = sizeof(VTFramesContext),
>>>         .device_create        = vt_device_create,
>>> +    .device_derive        = vt_device_derive,
>>>       .frames_hwctx_size    = sizeof(AVVTFramesContext),
>>>       .frames_init          = vt_frames_init,
>>>       .frames_get_buffer    = vt_get_buffer,
>>
>> This derivation behaviour doesn't make any sense inside libavutil.  Features which are only for the ffmpeg utility should be implemented inside the ffmpeg utility.
>>
>> (Also, try -init_hw_device.)
>>
 > Well this does make sense.
 >
 > If only one type of hardware device is in use, then `-init_hw_device` will suffice. However, if we use hardware filters with different device types, such as OpenCL, and we want to switch to a VideoToolbox filter later in the chain, we will have a issue. Since there is no `hwmap` between OpenCL and VideoToolbox, we need to perform a `hwdownload` followed by a `hwupload`. If the `derive_device` option cannot be used with `hwupload` due to `device_derive` not being implemented, the filter will struggle to find the correct device for uploading. It may attempt to upload to the OpenCL device without `device_derive` set if it comes after an OpenCL filter.
 >

Yes, it is unfortunate that the ffmpeg utility is missing this feature.

A libavfilter API user will set the correct hw_device_ctx on each filter and your use-case will work properly, but the ffmpeg utility does not fully expose the functionality of libavfilter to be able to do this.

This is mostly for syntax reasons; implementation would be straightforward.  If you can think of a good way to represent this in options to the ffmpeg utility (which doesn't become too horrible with escaping) then I would very much welcome suggestions.

Thanks,

- Mark