[FFmpeg-devel] [PATCH 10/14] swscale/range_convert: fix mpeg ranges in yuv range conversion for non-8-bit pixel formats

[Ramiro Polla]

On Tue, Sep 24, 2024 at 2:24 PM Michael Niedermayer
<michael at niedermayer.cc> wrote:
> On Mon, Sep 23, 2024 at 02:40:13PM +0200, Ramiro Polla wrote:
> > There is an issue with the constants used in YUV to YUV range conversion,
> > where the upper bound is not respected when converting to mpeg range.
> >
> > With this commit, the constants are calculated at runtime, depending on
> > the bit depth. This approach also allows us to more easily understand how
> > the constants are derived.
> >
> > For bit depths <= 14, the number of fixed point bits has been set to 14
> > for all conversions, to simplify the code.
> > For bit depths > 14, the number of fixed points bits has been raised and
> > set to 18, to allow for the conversion to be accurate enough for the mpeg
> > range to be respected.
> >
> > The convert functions now take the conversion constants (amax, coeff,
> > and offset) as function arguments.
> > For bit depths <= 14, amax is 16-bit and offset is 32-bit.
> > For bit depths > 14, amax is 32-bit and offset is 64-bit.
> >
> > NOTE: all simd optimizations for range_convert have been disabled.
> >       they will be re-enabled when they are fixed for each sub-arch.
> >
> > NOTE2: the same issue still exists in rgb2yuv conversions, which is not
> >        addressed in this commit.
> > ---
>
> This appears to break:
>
> make fate-filter-owdenoise-sample
> TEST    filter-owdenoise-sample
> stddev:12247.77 PSNR: 14.57 MAXDIFF:65280 bytes:   576000/   576000
> MAXDIFF: |65280 - 1| >= 3539
> Test filter-owdenoise-sample failed. Look at tests/data/fate/filter-owdenoise-sample.err for details.
> make: *** [tests/Makefile:312: fate-filter-owdenoise-sample] Error 1

This is weird. This error didn't show up when I was testing with make
fate. I ran the test individually and I saw the failure. Then I ran
the test again with GEN=1, there are no diffs in the source tree, and
now I can't get the test to fail anymore. I'll have to check again.

> What peformance impact do non constants have ?

It is mostly faster than using constants, both on x86_64 and aarch64.
I will include more comprehensive benchmarks in v2.

Also I figured out I could make it even faster by saturating the
output instead of limiting the range of the input. I'll send a new
patchset when this is done.

Ramiro