[FFmpeg-devel] [PATCH] avcodec/x86/hevc: fix luma 12b overflow

[Ronald S. Bultje]

Hi,

On Sun, Feb 25, 2024 at 3:28 AM J. Dekker <jdek at itanimul.li> wrote:

> Weak filter can overflow in delta0 calculation before >> 4 in int16.
>
> Signed-off-by: J. Dekker <jdek at itanimul.li>
> ---
>
>  I do not know x86 simd at all, so this is just an attempt to fix
>  the implementation rather than write extremely performant code.
>
>  Suggestions welcome.
>

https://pastebin.com/KvcbQ2nK

Most of this can remain in 16bits when doing pabsw before the add. The odd
thing is that if I break (intentionally) the sse2 version, the ssse3/avx
checkams tests start failing occasionally also. I'm not sure why that is
the case. Not sure how to measure perf impact, I don't think there is any
for the non-12bit case (same number of instructions) but I didn't test very
hard and the perf is not very stable...

Ronald