[FFmpeg-devel] GSoC

[Mark Thompson]

On 15/03/18 01:08, Dylan Fernando wrote:
> On Sun, Mar 11, 2018 at 10:18 PM, Mark Thompson <sw at jkqxz.net> wrote:
> 
>> On 11/03/18 04:36, Dylan Fernando wrote:
>>> On Thu, Mar 8, 2018 at 8:57 AM, Mark Thompson <sw at jkqxz.net> wrote:
>>>
>>>> On 07/03/18 03:56, Dylan Fernando wrote:
>>>>> Thanks, it works now
>>>>>
>>>>> Would trying to implement an OpenCL version of vf_fade be a good idea
>>>> for a
>>>>> qualification task, or would it be a better idea to try a different
>>>> filter?
>>>>
>>>> That sounds like a sensible choice to me, though if you haven't written
>> a
>>>> filter before you might find it helpful to write something simpler
>> first to
>>>> understand how it fits together (for example: vflip, which has trivial
>>>> processing parts but still needs the surrounding boilerplate).
>>>>
>>>> - Mark
>>>>
>>>> (PS: be aware that top-posting is generally frowned upon on this mailing
>>>> list.)
>>>>
>>>>
>>>>> On Wed, Mar 7, 2018 at 1:20 AM, Mark Thompson <sw at jkqxz.net> wrote:
>>>>>
>>>>>> On 06/03/18 12:37, Dylan Fernando wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am Dylan Fernando. I am a Computer Science student from Australia.
>> I
>>>> am
>>>>>>> new to FFmpeg and I wish to apply for GSoC this year.
>>>>>>> I would like to do the Video filtering with OpenCL project and I
>> have a
>>>>>> few
>>>>>>> questions. Would trying to implement an opencl version of vf_fade be
>> a
>>>>>> good
>>>>>>> idea for the qualification task, or would I be better off using a
>>>>>> different
>>>>>>> filter?
>>>>>>>
>>>>>>> Also, I’m having a bit of trouble with running unsharp_opencl. I
>> tried
>>>>>>> running:
>>>>>>> ffmpeg -hide_banner -nostats -v verbose -init_hw_device
>> opencl=ocl:0.1
>>>>>>> -filter_hw_device ocl -i space.mpg -filter_complex unsharp_opencl
>>>>>> output.mp4
>>>>>>>
>>>>>>> but I got the error:
>>>>>>> [AVHWDeviceContext @ 0x7fdac050c700] 0.1: Apple / Intel(R) Iris(TM)
>>>>>>> Graphics 6100
>>>>>>> [mpeg @ 0x7fdac3132600] max_analyze_duration 5000000 reached at
>> 5005000
>>>>>>> microseconds st:0
>>>>>>> Input #0, mpeg, from 'space.mpg':
>>>>>>>   Duration: 00:00:21.99, start: 0.387500, bitrate: 6108 kb/s
>>>>>>>     Stream #0:0[0x1e0]: Video: mpeg2video (Main), 1 reference frame,
>>>>>>> yuv420p(tv, bt470bg, bottom first, left), 720x480 [SAR 8:9 DAR 4:3],
>>>> 6000
>>>>>>> kb/s, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
>>>>>>> Stream mapping:
>>>>>>>   Stream #0:0 (mpeg2video) -> unsharp_opencl
>>>>>>>   unsharp_opencl -> Stream #0:0 (mpeg4)
>>>>>>> Press [q] to stop, [?] for help
>>>>>>> [graph 0 input from stream 0:0 @ 0x7fdac0418800] w:720 h:480
>>>>>> pixfmt:yuv420p
>>>>>>> tb:1/90000 fr:30000/1001 sar:8/9 sws_param:flags=2
>>>>>>> [auto_scaler_0 @ 0x7fdac05232c0] w:iw h:ih flags:'bilinear' interl:0
>>>>>>> [Parsed_unsharp_opencl_0 @ 0x7fdac0715a80] auto-inserting filter
>>>>>>> 'auto_scaler_0' between the filter 'graph 0 input from stream 0:0'
>> and
>>>>>> the
>>>>>>> filter 'Parsed_unsharp_opencl_0'
>>>>>>> Impossible to convert between the formats supported by the filter
>>>> 'graph
>>>>>> 0
>>>>>>> input from stream 0:0' and the filter 'auto_scaler_0'
>>>>>>> Error reinitializing filters!
>>>>>>> Failed to inject frame into filter network: Function not implemented
>>>>>>> Error while processing the decoded data for stream #0:0
>>>>>>> Conversion failed!
>>>>>>>
>>>>>>> How do I correctly run unsharp_opencl? Should I be running it on a
>>>>>>> different video file?
>>>>>>
>>>>>> It's intended to be used in filter graphs where much of the activity
>> is
>>>>>> already happening on the GPU, so the input and output are in the
>>>>>> AV_PIX_FMT_OPENCL format which contains GPU-side OpenCL images.
>>>>>>
>>>>>> If you want to use it standalone then you need hwupload and hwdownload
>>>>>> filters to move the frames between the CPU and GPU.  For your example,
>>>> it
>>>>>> should work with:
>>>>>>
>>>>>> ffmpeg -init_hw_device opencl=ocl:0.1 -filter_hw_device ocl -i
>> space.mpg
>>>>>> -filter_complex hwupload,unsharp_opencl,hwdownload output.mp4
>>>>>>
>>>>>> (There are constraints on what formats can be used and therefore
>>>> suitable
>>>>>> files (or required format conversions), but I believe a normal yuv420p
>>>>>> video like this should work in all cases.)
>>>>>>
>>>>>> - Mark
>>>>
>>>
>>> Thanks.
>>>
>>> How is AV_PIX_FMT_OPENCL formatted? When using read_imagef(), does xyzw
>>> correspond to RGBA respectively, or to YUV? Would I have to account for
>>> different formats? If so, how do I check the format of the input?
>>
>> See libavutil/hwcontext_opencl.c and in particular the functions
>> opencl_get_buffer(), opencl_pool_alloc() and opencl_get_plane_format() for
>> the code creating the AV_PIX_FMT_OPENCL images.
>>
>> It tries to support all formats which are representable as OpenCL images,
>> so the component values are dependent on what the format of the underlying
>> image is.  What can actually be represented does depends a bit on the
>> implementation - for example, CL_R channel order is needed for all planar
>> YUV images, and CL_RG is needed as well for NV12 and P010 support.  The
>> data_type is always UNORM_INT8 or UNORM_INT16 (depending on depth,
>> intermediate depths like 10-bit require are treated as UNORM_INT16 and
>> require an MSB-packed format like P010 rather than an LSB-packed format
>> like YUV420P), so it should always be read as a float (float2, float4) in
>> the CL kernels.
>>
>> Given that, if you have kernels which are not dependent on interactions
>> between components then you don't actually need to care about the
>> underlying format - use float4 everywhere and what's actually in xyzw
>> doesn't matter.  See the program_opencl examples <
>> http://ffmpeg.org/ffmpeg-filters.html#program_005fopencl-1> for some
>> cases of this, and the unsharp_opencl filter is also close to this (it only
>> cares about luma vs. chroma planes).
>>
>> If on the other hand you do need to know exactly where the components are,
>> then you will need to look at the sw_format of the incoming hw_frames_ctx
>> (it's on the input link at when the config_inputs function is called on a
>> filter input pad).  If you can't easily support all formats then rejecting
>> unsupported ones here with a suitable error message is fine (there isn't
>> currently any negotiation of that format, so it will be up to the user to
>> get it into the right state).  With only one or a small number of formats
>> there then you can know exactly what is in the xyzw components and
>> therefore use them however you like.
> 
> Thanks, that helps.
> 
> I need to know exactly what is in xyzw, since I’m trying to implement
> vf_fade, and the color parameter is given as rgba. I managed to get it
> working on yuv files. I’m currently trying to test it on different pixel
> formats. What other file types should I test my filter on? Is there some
> way I can download different video files with different pixel formats to
> test my filter on?

If you upload from the CPU then it is easiest to add "format=name" at the front of your filter chain to get swscale to do conversion so that you can test different formats (e.g. 'format=bgra,hwupload,...').

If you're mapping from something else then it is more difficult, because hardware video generally wants to use NV12 format everywhere.  With VAAPI you can use the scale_vaapi filter to convert the input to other formats before mapping to OpenCL ('scale_vaapi=format=bgra,hwmap=derive_device=opencl,...'), but I don't think it's possible in any of the other supported APIs.

- Mark