[FFmpeg-devel] [PATCH v2] lavfi/vf_libplacebo: add vulkan device import fallback

[Niklas Haas]

On Thu, 11 May 2023 14:01:34 +0200 Lynne <dev at lynne.ee> wrote:
> May 11, 2023, 13:21 by h.leppkes at gmail.com:
> 
> > On Thu, May 11, 2023 at 11:32 AM Lynne <dev at lynne.ee> wrote:
> >
> >>
> >> May 11, 2023, 10:39 by ffmpeg at haasn.xyz:
> >>
> >> > From: Niklas Haas <git at haasn.dev>
> >> >
> >> > Recent versions of libplacebo have required Vulkan versions incompatible
> >> > with lavu Vulkan hwcontexts. While this is expected to change
> >> > eventually, breaking vf_libplacebo every time there is such a transition
> >> > period is obviously undesired behavior, as the following sea of bug
> >> > reports shows.
> >> >
> >> > This commit adds a fallback path for pl_vulkan_import failures which
> >> > simply creates an internal device instead. Also useful when no interop
> >> > with lavu vulkan hwframes is needed or desired.
> >> >
> >> > Fixes: https://github.com/haasn/libplacebo/issues/170
> >> > Fixes: https://github.com/mpv-player/mpv/issues/9589#issuecomment-1535432185
> >> > Fixes: https://code.videolan.org/videolan/libplacebo/-/issues/270
> >> >
> >>
> >> NAK. The whole point of the hwcontext infrastructure is to be
> >> explicit, and creating a new device behind the scenes is anything but that.
> >>
> >
> > This is not a native vulkan filter, it is an external library not
> > married to our vulkan infrastructure - it merely has compatibility for
> > it.
> >
> > Ensuring it works, as it might have separate requirements or a
> > different development cycle, is hardly a bad thing. And it greatly
> > simplifies the usability for users that only want a quick GPU
> > processing.
> >
> 
> vf_libplacebo already creates its own device for software, vaapi and DRM
> frames, this patch is specifically for vulkan input frames, which have
> a vulkan device/frames context.

This is not true. Currently, if vf_libplacebo is not provided a working vulkan
device, it fails initializing entirely. Which raises an excellent point:

Without this patch, interop with vaapi etc. frames is not possible unless the
user *also* initializes a vulkan device *and* ensures it's passed to the
vf_libplacebo filter (e.g. via -filter_hw_device). And interop with non-vulkan
hardware filters is not possible at all, unless there's a way to specify
a different hwdevice for different filters?

> If the vulkan device with the frames is unusable, it is either intentional
> (if given by the API users) or a bug (if using a context created by lavu).
> Masking over which is not appropriate.

Your analysis omits an important third case:
./ffmpeg -i FILE.mp4 -vf libplacebo OUTPUT.mp4

One might be forgiven a naive user expecting this command to work as written.
But before this patch, it does not - one must *also* specify -init_hw_device
vulkan, even if not using any `_vulkan` filters! This is IMO very unexpected,
and most definitely (in all likelihood) not the user intent.