[FFmpeg-devel] [PATCH 2/2] swscale/output: Bias 16bps output calculations to improve non overflowing range for GBRP16/GBRPF32

[Michael Niedermayer]

On Wed, Nov 02, 2022 at 10:31:27PM +0100, Michael Niedermayer wrote:
> On Wed, Nov 02, 2022 at 10:16:57PM +0100, Andreas Rheinhardt wrote:
> > Michael Niedermayer:
> > > On Wed, Nov 02, 2022 at 10:02:39PM +0100, Michael Niedermayer wrote:
> > >> Fixes: integer overflow
> > >> Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> > >> ---
> > >>  libswscale/output.c       | 25 +++++++++++--------------
> > >>  libswscale/x86/output.asm | 16 +++++++++++++++-
> > >>  2 files changed, 26 insertions(+), 15 deletions(-)
> > > 
> > > Note, these corner case overflows could affect some of the similary
> > > implemented cases that are not depth 16 too
> > > 
> > > ill fix them if issues are replicated
> > > 
> > 
> > The checkasm-sw_gbrp runs into many overflows (when run under UBSan);
> > e.g. fate.ffmpeg.org tells me of an issue in line 2289. Said line is not
> > touched in your commits.
> 
> checkasm-sw_gbrp feeds random data widely outside sane ranges in.
> the test certainly makes no sense for testing asm. There is no point
> in matching C for widely invalid cases. Of cousre we shouldnt overflow
> if any of this can be triggered with valid and real input (which probably
> can be done in some cases)

also to elaborate further, checkasm-sw_gbrp sets everything randomly
that includes yuv2rgb matrix values and bits outside the valid input sample
range and the resample filter coeffs.

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

If a bugfix only changes things apparently unrelated to the bug with no
further explanation, that is a good sign that the bugfix is wrong.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20221102/fb7c170d/attachment.sig>