[FFmpeg-devel] [PATCH] swscale/yuv2rgb: fix shift values for conversion to X2RGB10

[Manuel Stoeckl]

> On Mon, Sep 20, 2021 at 10:21:23PM -0400, Manuel Stoeckl wrote:
> > This resolves a problem where conversions from YUV to X2RGB10LE
> > would produce color values a factor 4 too small.
> > 
> > The variable 'yval' used later in the switch case 30 has range
> > [0,255], but the color channel values in X2RGB10 have two more
> > bits than 'yval' and can go up to 1023. Increasing (r|g|b)base
> > by 2 effectively multiplies yval by 4 and fixes this discrepancy.  
> 
> does white have 1023 ? or 1020 ?
> a multiplication by 4 would not in general produce a correctly scaled
> 10bit per channel value from 8bit

I've replied to this email with a new version of the patch set, that
now ensures the channel values in the YUV->RGB table created in
ff_yuv2rgb_c_init_tables go up to 1023.

However, this does not mean the code will always convert 1.0 white to
1.0 white; for example, pure 0xffff RGBA64 white gets converted to
X2RGB10 grey level 1019. Presumably there is an off-by-one rounding
error somewhere in the RGB->YUV->RGB conversion path, but I don't think
tracking it down is worth it at the moment.