[FFmpeg-devel] [PATCH 4/4] avcodec/osq: avoid using too large numbers for shifts and integers in update_residue_parameter()

[Michael Niedermayer]

On Fri, Sep 15, 2023 at 03:54:19PM +0200, Paul B Mahol wrote:
> On Fri, Sep 15, 2023 at 3:12 PM Michael Niedermayer <michael at niedermayer.cc>
> wrote:
> 
> > The code should be changed to not use floats in the VLC parameters
> > This patch merely fixes undefined behavior
> >
> > Fixes: 2.96539e+09 is outside the range of representable values of type
> > 'int'
> > Fixes: Assertion n>=0 && n<=32 failed at libavcodec/get_bits.h:423
> > Fixes:
> > 62241/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_OSQ_fuzzer-4525761925873664
> >
> 
> 
> NAK
> 
> Breaks decoding.

Please provide sample that works before and fails after this.

That said, it has to be pointed out that the current code in osq is
buggy in multiply ways. The use of floats for computing vlc parameters is
not portable (unless theres alot of luck).

thx


[....]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

I have often repented speaking, but never of holding my tongue.
-- Xenocrates
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20230915/b9532360/attachment.sig>