[FFmpeg-devel] SWS cleanup / SPI Funding Suggestion

[Michael Niedermayer]

Hi

On Sat, Oct 14, 2023 at 07:24:51PM +0200, Niklas Haas wrote:
> On Sat, 14 Oct 2023 19:00:36 +0200 Michael Niedermayer <michael at niedermayer.cc> wrote:
> > Well there are 2 further aspects with that.
> > 
> > The first one is bluntly put. If you dont understand the old code, then
> > you probably are not qualified to write better code.
> > People tend not to successfully improve things they dont understand.
> 
> I have a deep understanding of colorspaces and the necessary conversion
> steps between them, and am also in a good position to integrate
> libplacebo as a possible backend.
> 
> However, I do not have a good understanding of CPU/SIMD code, nor the
> various swscale internals, beyond the cursory investigation I needed for
> some recent swscale bugs I encountered. So I'll definitely need some
> help along the way to fully understand those swscale internals.

iam happy to help., as far as i remember (and as far as i know) things.
Also ronald probably knows quite a bit from the older parts.
various people also added bits over the years, they can probably help
within these bits. And we had some GSoC contributions from Pedro Arthur,
he certainly is worth a try to ask if you have a question related to his
code.


> 
> > The 2nd issue is, ATM, i maintain swscale. If iam involved in the new
> > effort and understand it either because of that or because it has some
> > similarity then i can continue to maintain swscale. If its totally
> > different and i was totally not involded then i also will not maintain
> > it obviously.
> 
> I think it would be possible to join forces to the extent needed to
> arrive at a satisfactory result. At the very least, I have very limited
> experience working with "irregular" packed formats.
> 
> Obviously, my intent is not to blanket discard the swscale internals. It
> has years and years of optimized kernels for various platforms just
> lying around, wanting to be used. Hence my proposal of redesigning the
> high-level logic, rather than the low-level details.

yes, i expect that my role in this will probably be limited to awnsering
questions, reviewing code and maybe running some tests.
Not sure i will have the motivation to do much heavy restructuring of things
It might be that i suddenly see something i want to work on ;) but besides this
iam happy to leave it to you and others


> 
> > This is something to be especially aware of in case the cleanup/new
> > code would be done by someone who comes, does it and leaves. you
> > could end up with nicer code thats then unmaintained.
> > 
> > PS: whats the real issue with sws ?
> > it evolved out of a piece yuv->rgb converter from a video player.
> > It evolved from that and stuff was added into it.
> > This is a similar situation to why ffmpeg.c needed cleanup
> 
> Yes, it amounts to the usual disentangling of various special cases and
> branches into one top-down control flow that knows about all of these
> special cases and fast/slow paths to begin with.
> 
> My goal is to arrive at a place where we have one single code flow that
> looks something like:
> 
> 1. Settle the complete descriptions of the source and destination format/csp
> 2. Establish a list of operations to get from A to B, taking into
>    account user settings
> 3. Determine and dispatch the best available functions for each operation

I think this is the obvious logic way to design it, yes


> 
> With the necessary code separation and/or layers of abstraction in place
> to make this design manageable. In particular, steps 1 and 2 should be
> expanded to include things like conversion between primaries, conversion
> between HDR and SDR, conversion between YUV/RGB, and so on.

One thing that will be needed is to allow alternatives for seperation
for example an optimized codepath might do vertical scaling and convertion
to BGR16 in one piece. While both operations also should probably be available
individually.
This adds complexity but such combined operations are used in the current
codepathes so if the existing optimizations are supported then the current
way things are broken up into steps need to be supported


> 
> In particular, I also want to eventually add the ability to plug "Apply
> a 3DLUT" in as a possible operation type for colorspace conversion,
> probably by sharing the code that is already written for vf_lut3d.

Speaking of this, the horizontal and vertical scaling is basically a
FIR filter being applied. Which is also why swscale allows the user
to provide such filters, they can be applied by just using the already
needed codepathes with no extra complexity

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Breaking DRM is a little like attempting to break through a door even
though the window is wide open and the only thing in the house is a bunch
of things you dont want and which you would get tomorrow for free anyway
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20231015/baf734be/attachment.sig>