[FFmpeg-devel] [RFC] New swscale internal design prototype

[Niklas Haas]

On Sun, 09 Mar 2025 11:18:04 -0700 Rémi Denis-Courmont <remi at remlab.net> wrote:
> Hi,
> 
> Le 8 mars 2025 14:53:42 GMT-08:00, Niklas Haas <ffmpeg at haasn.xyz> a écrit :
> >https://github.com/haasn/FFmpeg/blob/swscale3/doc/swscale-v2.txt
> 
> >I have spent the past week or so ironing 
> >I wanted to post it here to gather some feedback on the approach. Where does
> >it fall on the "madness" scale? Is the new operations and optimizer design
> >comprehensible? Am I trying too hard to reinvent compilers? Are there any
> >platforms where the high number of function calls per frame would be
> >probitively expensive? What are the thoughts on the float-first approach? See
> >also the list of limitations and improvement ideas at the bottom of my design
> >document.
> 
> Using floats internally may be fine if there's (almost) never any spillage, but that necessarily implies custom calling conventions. And won't work with as many as 32 pixels. On RVV 128-bit, you'd have only 4 vectors. On Arm NEON, it would be even worse as scalars/constants need to be stored in vectors as well.

I think that a custom calling convention is not as unreasonable as it may sound,
and will actually be easier to implement than the standard calling convention
since functions will not have to deal with pixel load/store, nor will there be
any need for "fused" versions of operations (whose only purpose is to avoid
the roundtrip through L1).

The pixel chunk size is easily changed; it is a compile time constant and there
are no strict requirements on it. If RISC-V (or any other platform) struggles
with storing 32 floats in vector registers, we could go down to 16 (or even 8);
the number 32 was merely chosen by benchmarking and not through any careful
design consideration.

During my initial prototype, I was using 16 pixel chunks (= 512 bits, or
enough to fit into an m4 register on RVV 128). That still gives you room to
keep 4 vectors in memory (for the custom CC) and still have 4 spare vectors to
implement operations. I *think* that should be sufficient, with the current
set of vector operations.

> 
> Otherwise transferring two or even four times as much data to/from memory at every step is probably going to more than absorb any performance gains from using floats (notably not needing to scale values).

It's quite possible. I don't think that there is any major barrier to adding
fixed precision integer support to SwsOp design *per se*. The main reason
I am hesitant to explore that option comes from the fact that I don't want to
introduce (or encourage) platform-specific variations in the output. By forcing
all platforms to adhere to a single precision, we can guarantee a consistent
output regardless of the optimization decisions.

So it would probably have to involve us switching from floats to fixed16
across the board, even on x86.

In either case, I think I will stick with the float centric design during the
development of my prototype if for no other reason than simplicity, unless
there is a vary major performance issue associated with them.

Do you have access to anything with decent RVV F32 support that we could use
for testing? It's my understanding that existing RVV implementations have been
rather primitive.

> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".