[FFmpeg-devel] [PATCH 1/5] avcodec/apedec: Fix 32bit int overflow in do_apply_filter()

[Tomas Härdin]

tis 2019-09-03 klockan 13:25 +0200 skrev Michael Niedermayer:
> On Tue, Sep 03, 2019 at 12:16:47PM +0200, Tomas Härdin wrote:
> > tis 2019-09-03 klockan 02:14 +0200 skrev Michael Niedermayer:
> > > Fixes: signed integer overflow: 2147480546 + 4096 cannot be represented in type 'int'
> > > Fixes: 16280/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-5123442566758400
> > > 
> > > Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
> > > Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> > > ---
> > >  libavcodec/apedec.c | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/libavcodec/apedec.c b/libavcodec/apedec.c
> > > index 490b11b94e..ed22f0f019 100644
> > > --- a/libavcodec/apedec.c
> > > +++ b/libavcodec/apedec.c
> > > @@ -1266,7 +1266,7 @@ static void do_apply_filter(APEContext *ctx, int version, APEFilter *f,
> > >                                                       f->delay - order,
> > >                                                       f->adaptcoeffs - order,
> > >                                                       order, APESIGN(*data));
> > > -        res = (res + (1 << (fracbits - 1))) >> fracbits;
> > > +        res = (int)(res + (1U << (fracbits - 1))) >> fracbits;
> > 
> > Does this mean the old code was decoding incorrectly, or is this just
> > an UB fix? (fixing UBs is good of course)
> 
> it is just a UB fix.
> If iam not misreading the reference source then it has the same UB in it
> so there is no correct way in case this ever triggers on a real file.
> 
> The specific patch here was done so that the result should hopefully
> match what a common machine with 32bit int (most platforms) would
> likely produce.

Right, and the cast back to int is required for proper sign extension
in the >> fracbits. Looks OK to me then. Kind of a weird codec feature
IMO

/Tomas