[FFmpeg-devel] [PATCH] swscale/slice: fix init of 32 bpc planes

[Michael Niedermayer]

Hi Niklas

On Wed, Dec 11, 2024 at 09:25:12AM +0100, Niklas Haas wrote:
> From: Niklas Haas <git at haasn.dev>
> 
> In input.c and output.c and many other places, swscale follows the rule of using
> 15-bit intermediate if output bpc is <= 8, and 19-bit (inside int32_t)
> intermediate otherwise. See e.g. the comments on hyScale() on
> swscale_internal.h. These are also the coefficients that yuv2gbrpf32_full_X_c()
> is using.
> 
> In contrast to this, the plane init code in slice.c (function fill_ones) is
> assuming that we use 35-bit intermediates (inside 64-bit integers) for this
> case, seemingly added by commit b4967fc71c63eae8cd96f9c46cd3e1fbd705bbf9 with
> no further justification.
> 
> This causes a mismatch whenever the implicitly initialized plane contents leak
> out to the output, e.g. when converting from grayscale to RGB.
> 
> Fixes: ticket #10716
> Signed-off-by: Niklas Haas <git at haasn.dev>
> Sponsored-by: Sovereign Tech Fund
> ---
>  libswscale/slice.c | 6 +-----
>  1 file changed, 1 insertion(+), 5 deletions(-)

ultimately 32bit on teh input or output side require more than 32bit
internally to maintain precission.

if this patch makes it all match up, its ok for now but 18bit dont seem
enough for 32bit data

thx


[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

What is money laundering? Its paying someone and not telling the government.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20241216/46ec5fb0/attachment.sig>