[FFmpeg-devel] [PATCH] avfilter: add v360_vulkan filter

Paul B Mahol onemda at gmail.com
Fri Jul 5 22:04:48 EEST 2024 

On Fri, Jul 5, 2024 at 5:11 PM TADANO Tokumei <aimingoff at pc.nifty.jp> wrote:

>
> On 2024/07/05 0:03, Paul B Mahol wrote:
> > On Thu, Jul 4, 2024 at 4:48 PM TADANO Tokumei <aimingoff at pc.nifty.jp>
> wrote:
> >
> >>
> >> On 2024/06/25 22:27, Paul B Mahol wrote:
> >>> On Tue, Jun 25, 2024 at 3:17 PM Dennis Mungai <dmngaie at gmail.com>
> wrote:
> >>>
> >>>> On Tue, 25 Jun 2024 at 16:07, Paul B Mahol <onemda at gmail.com> wrote:
> >>>>
> >>>>> On Tue, Jun 25, 2024 at 2:47 PM Dennis Mungai <dmngaie at gmail.com>
> >> wrote:
> >>>>>
> >>>>>> On Sun, 28 Jun 2020 at 16:59, James Almer <jamrial at gmail.com>
> wrote:
> >>>>>>
> >>>>>>> On 6/27/2020 7:54 AM, Paul B Mahol wrote:
> >>>>>>>> On 6/26/20, James Almer <jamrial at gmail.com> wrote:
> >>>>>>>>> On 6/26/2020 4:16 PM, Lynne wrote:
> >>>>>>>>>> Jun 26, 2020, 19:05 by jamrial at gmail.com:
> >>>>>>>>>>
> >>>>>>>>>>> On 6/26/2020 1:58 PM, Paul B Mahol wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> On 6/26/20, James Almer <jamrial at gmail.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> On 6/26/2020 1:13 PM, Paul B Mahol wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On 6/26/20, Paul B Mahol <onemda at gmail.com> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On 6/24/20, Paul B Mahol <onemda at gmail.com> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Signed-off-by: Paul B Mahol <onemda at gmail.com>
> >>>>>>>>>>>>>>>> ---
> >>>>>>>>>>>>>>>> Please test it and reports benchmark results on non toyish
> >> GPUs.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Please, I really need it!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Why nobody wants to help?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> This makes me really really sad.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I can try. Can you give an example command line with a
> publicly
> >>>>>>>>>>>>> available sample?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> You can use any sample, samples are irrelevant. just report
> resolution
> >>>>>>>>>>>> used.
> >>>>>>>>>>>> Also compare with CPU solution with just v360 filter.
> >>>>>>>>>>>>
> >>>>>>>>>>>> For yuv420p inputs:
> >>>>>>>>>>>>
> >>>>>>>>>>>> ffmpeg -init_hw_device vulkan=vulkan -i INPUT
> -filter_hw_device vulkan
> >>>>>>>>>>>> -vf
> hwupload,v360_vulkan=fisheye:e:ih_fov=180:iv_fov=180:w=2048:h=1024,hwdownload,format=yuv420p
> -f null -
> >>>>>>>>>>>>
> >>>>>>>>>>>> ffmpeg -i INPUT -vf
> v360=fisheye:e:ih_fov=180:iv_fov=180:w=2048:h=1024 -f null -
> >>>>>>>>>>>>
> >>>>>>>>>>>> Just make sure that you get same output resolution in both
> cases. If
> >>>>>>>>>>>> not, adjust w/h accordingly.
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I'm getting a segfault on vkGetInstanceProcAddr, and i don't
> know if
> >>>>>>>>>>> it's your filter or some issue in the driver. gdb is not being
> helpful.
> >>>>>>>>>>> _______________________________________________
> >>>>>>>>>>> ffmpeg-devel mailing list
> >>>>>>>>>>> ffmpeg-devel at ffmpeg.org
> >>>>>>>>>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >>>>>>>>>>>
> >>>>>>>>>>> To unsubscribe, visit link above, or email
> >>>>>>>>>>> ffmpeg-devel-request at ffmpeg.org>  with subject "unsubscribe".
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Can you test again but replace EXT_EXTERNAL_HOST_MEMORY in
> >>>>>>>>>> optional_device_exts
> >>>>>>>>>> with EXT_NO_FLAG?
> >>>>>>>>>
> >>>>>>>>> Applying
> >>>>>>>>>
> >>>>>>>>>> diff --git a/libavutil/hwcontext_vulkan.c
> b/libavutil/hwcontext_vulkan.c
> >>>>>>>>>> index 5e51d0390f..49fb7827ba 100644
> >>>>>>>>>> --- a/libavutil/hwcontext_vulkan.c
> >>>>>>>>>> +++ b/libavutil/hwcontext_vulkan.c
> >>>>>>>>>> @@ -228,7 +228,7 @@ static const VulkanOptExtension
> optional_device_exts[]
> >>>>>>>>>> = {
> >>>>>>>>>>        { VK_EXT_EXTERNAL_MEMORY_DMA_BUF_EXTENSION_NAME,
> >>>>>>>>>> EXT_EXTERNAL_DMABUF_MEMORY, },
> >>>>>>>>>>        { VK_EXT_IMAGE_DRM_FORMAT_MODIFIER_EXTENSION_NAME,
> >>>>>>>>>> EXT_DRM_MODIFIER_FLAGS,     },
> >>>>>>>>>>        { VK_KHR_EXTERNAL_SEMAPHORE_FD_EXTENSION_NAME,
> >>>>>>>>>> EXT_EXTERNAL_FD_SEM,        },
> >>>>>>>>>> -    { VK_EXT_EXTERNAL_MEMORY_HOST_EXTENSION_NAME,
> >>>>>>>>>> EXT_EXTERNAL_HOST_MEMORY,   },
> >>>>>>>>>> +    { VK_EXT_EXTERNAL_MEMORY_HOST_EXTENSION_NAME,
> >>>>>>>>>> EXT_NO_FLAG,   },
> >>>>>>>>>>    };
> >>>>>>>>>>
> >>>>>>>>>>    /* Converts return values to strings */
> >>>>>>>>>
> >>>>>>>>> I still get a segfault:
> >>>>>>>>>
> >>>>>>>>>> Thread 1 received signal SIGSEGV, Segmentation fault.
> >>>>>>>>>> 0x00007ff8ed9274c0 in vkGetInstanceProcAddr ()
> >>>>>>>>>>      from
> >>>>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> C:\Windows\System32\DriverStore\FileRepository\nv_dispi.inf_amd64_ccad5caddc3a3d35\nvoglv64.dll
> >>>>>>>>>
> >>>>>>>>> Debug output shows
> >>>>>>>>>
> >>>>>>>>>> [AVHWDeviceContext @ 000001532a762800] GPU listing:
> >>>>>>>>>> [AVHWDeviceContext @ 000001532a762800]     0: GeForce GTX 1660
> (discrete)
> >>>>>>>>>> (0x2184)
> >>>>>>>>>> [AVHWDeviceContext @ 000001532a762800] Using queue family 0
> (total queues:
> >>>>>>>>>> 16) for graphics
> >>>>>>>>>> [AVHWDeviceContext @ 000001532a762800]     QF 0 flags:
> (graphics)
> >>>>>>>>>> (compute) (transfers) (sparse)
> >>>>>>>>>> [AVHWDeviceContext @ 000001532a762800] Using queue family 2
> (total queues:
> >>>>>>>>>> 8) for compute
> >>>>>>>>>> [AVHWDeviceContext @ 000001532a762800]     QF 2 flags: (compute)
> >>>>>>>>>> (transfers) (sparse)
> >>>>>>>>>> [AVHWDeviceContext @ 000001532a762800] Using queue family 1
> (total queues:
> >>>>>>>>>> 2) for transfers
> >>>>>>>>>> [AVHWDeviceContext @ 000001532a762800]     QF 1 flags:
> (transfers)
> >>>>>>>>>> (sparse)
> >>>>>>>>>> [AVHWDeviceContext @ 000001532a762800] Using device extension
> >>>>>>>>>> "VK_EXT_external_memory_host"
> >>>>>>>>>> [AVHWDeviceContext @ 000001532a762800] Using device extension
> >>>>>>>>>> VK_EXT_external_memory_host
> >>>>>>>>>> [AVHWDeviceContext @ 000001532a762800] Using device: GeForce
> GTX 1660
> >>>>>>>>>> [AVHWDeviceContext @ 000001532a762800] Alignments:
> >>>>>>>>>> [AVHWDeviceContext @ 000001532a762800]
> >>>>>>>>>> optimalBufferCopyRowPitchAlignment: 1
> >>>>>>>>>> [AVHWDeviceContext @ 000001532a762800]
>  minMemoryMapAlignment:      64
> >>>>>>>>>
> >>>>>>>>> And crashes right after printing:
> >>>>>>>>>
> >>>>>>>>>> [Parsed_v360_vulkan_1 @ 000001532abd5c80] Shader v360_compute
> linked!
> >>>>>>>>>> Size: 5508 bytes
> >>>>>>>>>
> >>>>>>>>> I'm using the latest driver, 451.48, on Windows 10 2004.
> >>>>>>>>
> >>>>>>>> So other vulkan filters, like vulkan_chromaber works for you?
> >>>>>>>
> >>>>>>> All fail the same way in vkGetInstanceProcAddr, so the issue is in
> >>>>>>> hwcontext.
> >>>>>>>
> >>>>>>> Lynne should be able to test/bench this filter.
> >>>>>>>
> >>>>>>
> >>>>>>    Hello Paul, et al,
> >>>>>>
> >>>>>> Any updates on this filter?
> >>>>>>
> >>>>>
> >>>>> I'm no longer working on FFmpeg.
> >>>>>
> >>>>> The v360_opencl filter got in more useful state but due complete
> lack of
> >>>>> interest in it from users, I forgot about its existence.
> >>>>> The v360_vulkan filter is probably in worse shape than v360_opencl
> filter
> >>>>> and 0/negative interest, and I forgot about its existence too.
> >>>>>
> >>>>
> >>>> Is there an FFmpeg tree with v360_opencl (or a similar filter) out
> there?
> >>>>
> >>>
> >>>
> >>> I can give you patches/files for both v360 vulkan and opencl filter if
> you
> >>> want to hack on it.
> >>> But I will not adapt them for free to current library versions, nor
> provide
> >>> any kind of free support for it.
> >>>
> >>
> >> I tested v360_opencl filter with slight modification to your patch.
> >> It works, but it generates different output from v360 filter.
> >>
> >> For example, I tested a sample GoPro Max 360 video on:
> >>
> https://drive.google.com/drive/folders/1DlP6FVw-O6AIm6O0nSsC-mIoCrEBtmdG
> >>
> >> ffmpeg -i GS010010.360 -init_hw_device opencl:0.0 -vf
> >> "hwupload,v360_opencl=yaw=90,hwdownload,format=yuvj420p"
> output-v360cl.mkv
> >> ffmpeg -i GS010010.360 -vf v360=yaw=90 output-v360.mkv
> >>
> >> I'm not sure which is correct, but I feel the output of v360_opencl
> filter
> >> is better.
> >>
> >
> > Make sure that difference is not because you used yuvj420p pixel format
> at
> > end.
> > The output should be very similar - minus rounding, with visually no
> > difference.
> > Also why you transcode?
>
> Without specifying format parameter, the ffmpeg command would fails:
>
> $ ffmpeg -i GS010010.360 -init_hw_device opencl:0.0 -vf
> "hwupload,v360_opencl=yaw=90,hwdownload" output-v360cl.mkv
> ...
> Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'GS010010.360':
> ...
>     Stream #0:0[0x1](eng): Video: hevc (Main) (hvc1 / 0x31637668),
> yuvj420p(pc, bt709), 4096x1344 [SAR 1:1 DAR 64:21], 29970 kb/s, 29.97 fps,
> 29.97 tbr, 90k tbn (default)
> ...
> [hwdownload @ 0x7ef9e4005680] Invalid output format gray for hwframe
> download.
> [Parsed_hwdownload_2 @ 0x7ef9e4005540] Failed to configure output pad on
> Parsed_hwdownload_2
> [vf#0:0 @ 0x5621681baf00] Error reinitializing filters!
> [vf#0:0 @ 0x5621681baf00] Task finished with error code: -22 (Invalid
> argument)
> [vf#0:0 @ 0x5621681baf00] Terminating thread with return code -22 (Invalid
> argument)
> [vost#0:0/libx264 @ 0x5621680cf680] Could not open encoder before EOF
> [vost#0:0/libx264 @ 0x5621680cf680] Task finished with error code: -22
> (Invalid argument)
> [vost#0:0/libx264 @ 0x5621680cf680] Terminating thread with return code
> -22 (Invalid argument)
> [out#0/matroska @ 0x562167eac000] Nothing was written into output file,
> because at least one of its streams received no packets.
>
> The yuvj420p is same as input.
> I didn't intended to transcode.
>
> I'm not sure if this ML accepts attachments, but I attach (resized)
> snapshots.
>

Both outputs are garbage. Just that v360 by default converts from
equirectangular to cube 3x2, and v360_opencl does not support that one.

Dunno what projection you are trying to transfer from whatever product you
want to work with.

As I said already not going to help you with this for free.
Developers are also humans.


> >
> >>
> >>>
> >>>> A GPU-accelerated v360 filter is definitely useful to test and
> >> implement.
> >>>> Definitely more useful than the DNN stack in FFmpeg (by leaps and
> >> bounds)
> >>>> as far as applicability is concerned.
> >>>> _______________________________________________
> >>>> ffmpeg-devel mailing list
> >>>> ffmpeg-devel at ffmpeg.org
> >>>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >>>>
> >>>> To unsubscribe, visit link above, or email
> >>>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> >>>>
> >>> _______________________________________________
> >>> ffmpeg-devel mailing list
> >>> ffmpeg-devel at ffmpeg.org
> >>> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >>>
> >>> To unsubscribe, visit link above, or email
> >>> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> >> _______________________________________________
> >> ffmpeg-devel mailing list
> >> ffmpeg-devel at ffmpeg.org
> >> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >>
> >> To unsubscribe, visit link above, or email
> >> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
> >>
> > _______________________________________________
> > ffmpeg-devel mailing list
> > ffmpeg-devel at ffmpeg.org
> > https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> >
> > To unsubscribe, visit link above, or email
> > ffmpeg-devel-request at ffmpeg.org with subject
> "unsubscribe"._______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>

More information about the ffmpeg-devel mailing list