[FFmpeg-devel] [PATCH] swscale/input: Use unsigned intermediates in rgb64ToUV_c_template

[Michael Niedermayer]

On Tue, Nov 15, 2022 at 02:30:23PM -0500, Jeremy Dorfman wrote:
> Large rgb2yuv tables and high pixel values cause the intermediate
> int32_t of ru*r + gu*g + bu*b to exceed INT_MAX, which is undefined
> behavior. This causes libswscale built with LLVM -fsanitize=undefined to
> assert. Using unsigned integers instead has defined behavior and
> produces identical results, and makes rgb64ToUV_c_template match
> rgb64ToY_c_template.
> 
> Fixes: signed integer overflow
> ---
>  libswscale/input.c | 6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)

There are more cases which should be affected by this too.
I sent a more complete patchset

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Observe your enemies, for they first find out your faults. -- Antisthenes
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20221115/827f0c57/attachment.sig>