[FFmpeg-devel] [PATCH 2/2] swscale/output: Bias 16bps output calculations to improve non overflowing range for GBRP16/GBRPF32

[Martin Storsjö]

On Wed, 2 Nov 2022, Michael Niedermayer wrote:

> On Wed, Nov 02, 2022 at 10:16:57PM +0100, Andreas Rheinhardt wrote:
>> Michael Niedermayer:
>>> On Wed, Nov 02, 2022 at 10:02:39PM +0100, Michael Niedermayer wrote:
>>>> Fixes: integer overflow
>>>> Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
>>>> ---
>>>>  libswscale/output.c       | 25 +++++++++++--------------
>>>>  libswscale/x86/output.asm | 16 +++++++++++++++-
>>>>  2 files changed, 26 insertions(+), 15 deletions(-)
>>>
>>> Note, these corner case overflows could affect some of the similary
>>> implemented cases that are not depth 16 too
>>>
>>> ill fix them if issues are replicated
>>>
>>
>> The checkasm-sw_gbrp runs into many overflows (when run under UBSan);
>> e.g. fate.ffmpeg.org tells me of an issue in line 2289. Said line is not
>> touched in your commits.
>
> checkasm-sw_gbrp feeds random data widely outside sane ranges in.
> the test certainly makes no sense for testing asm. There is no point
> in matching C for widely invalid cases. Of cousre we shouldnt overflow
> if any of this can be triggered with valid and real input (which probably
> can be done in some cases)

Patches for the checkasm tests, to make them produce sane input values in 
the correct ranges, are very much welcome!

// Martin