[FFmpeg-devel] [PATCH] swscale/yuv2rgb: fix shift values for conversion to X2RGB10

[Manuel Stoeckl]

> > 
> > The variable 'yval' used later in the switch case 30 has range
> > [0,255], but the color channel values in X2RGB10 have two more
> > bits than 'yval' and can go up to 1023. Increasing (r|g|b)base
> > by 2 effectively multiplies yval by 4 and fixes this discrepancy.  
> 
> does white have 1023 ? or 1020 ?
> a multiplication by 4 would not in general produce a correctly scaled
> 10bit per channel value from 8bit

Yes, the shift approach is flawed in that it converts 255 to 1020.
Unfortunately, 255 does not divide 1023, so no matter what there will
be inputs where the conversion is slightly off.

I'd prefer not to optimize this conversion here, when the code will
inevitably need to be rewritten later to properly handle conversions
from 10-bit or higher YUV input. (Unfortunately, while I am certain
this is possible, I don't yet understand how the swscale code works
enough yet well enough to do that. There are also possible problems
with gamma and dithering to consider.)
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20210922/622500ed/attachment.sig>