[FFmpeg-devel] swscale : add bitexact conv for grayf32 and gray16 to f32 conv

[Martin Vignali]

> also, have you tried adding a small constant to tmp ?
> i would expect that this or a similar operation would allow moving
> away from all "unstable" points without really changing the output in a
> relevant way.
>
>
Can't find an op, in mult mode, in order to not raise the assert
But if i use add calc instead of mult,
I can have a bigger tolerance without raising the assert

In 8 bits mode
Tolerance
0.0000000000001 : pass
0.000000000001 : fail for i = 1
0.00000000001 : fail for i in [1, 2, 4, 8]
0.0000000001 : fail for i in [1, 2, 4, 8, 16, 32, 64]

In 16 bits mode
0.0000000000001 : pass
0.000000000001 : fail for i in [1, 257, 259, 261, 263, 265, 267, 269, 271,
273, 275, 277, 279, 281, 65407]

using this code :
#define assert_stable_float(x) av_assert0((float)(x+0.0000000000001) ==
(float)(x-0.0000000000001))
static void inline fill_uint_to_float_lut(SwsContext *c, int bitdepth) {
    static const double float_mult8 = 1.0 / 255.0;
    static const double float_mult16 = 1.0 / 65535.0;
    int i;
    double tmp = 0.;

    if (bitdepth == 8) { /*! fill uint8 to float lut */
        for (i = 0; i < 256; ++i){
            c->uint2float_lut[i] = (float)tmp;
            tmp += float_mult8;
            assert_stable_float(tmp);
        }
    } else if (bitdepth == 16) { /*! fill uint16 to float lut */
        for (i = 0; i < 65536; ++i){
            c->uint2float_lut[i] = (float)tmp;
            tmp += float_mult16;
            assert_stable_float(tmp);
        }
    } else { /*! unsupported bitdepth */
        av_assert0(0);
    }
}


Do you think 0.0000000000001, is enough for stable result in each supported
platform ?

Martin