[FFmpeg-devel] [PATCH 2/4] avcodec/apedec: Use 64bit in 4/3 calculation in do_apply_filter()

[Michael Niedermayer]

On Mon, Dec 06, 2021 at 03:50:34PM +0100, Anton Khirnov wrote:
> Quoting Michael Niedermayer (2021-12-06 12:01:09)
> > On Mon, Dec 06, 2021 at 11:23:12AM +0100, Anton Khirnov wrote:
> > > Quoting Michael Niedermayer (2021-12-03 18:19:54)
> > > > Fixes: Integer overflow
> > > > Fixes: 40973/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APE_fuzzer-6739312704618496
> > > > 
> > > > Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
> > > > Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> > > > ---
> > > >  libavcodec/apedec.c | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > 
> > > > diff --git a/libavcodec/apedec.c b/libavcodec/apedec.c
> > > > index 9c723f29977..5c3261cf5cb 100644
> > > > --- a/libavcodec/apedec.c
> > > > +++ b/libavcodec/apedec.c
> > > > @@ -1337,7 +1337,7 @@ static void do_apply_filter(APEContext *ctx, int version, APEFilter *f,
> > > >              absres = FFABSU(res);
> > > >              if (absres)
> > > >                  *f->adaptcoeffs = APESIGN(res) *
> > > > -                                  (8 << ((absres > f->avg * 3LL) + (absres > (f->avg + f->avg / 3))));
> > > > +                                  (8 << ((absres > f->avg * 3LL) + (absres > ((int64_t)f->avg + f->avg / 3))));
> > > >                  /* equivalent to the following code
> > > >                      if (absres <= f->avg * 4 / 3)
> > > >                          *f->adaptcoeffs = APESIGN(res) * 8;
> > > > -- 
> > > > 2.17.1
> > > 
> > > Does this not assume a 32bit int?
> > 
> > hmm
> > you mean avg could overflow 64bit ?
> > iam not sure that can happen but
> > we could make avg int32 or uint32 ?
> 
> Seems it cannot, but it's not completely obvious. So changing it to
> uint32 might make sense, since it cannot be negative.

ok will change it to uint32

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

When the tyrant has disposed of foreign enemies by conquest or treaty, and
there is nothing more to fear from them, then he is always stirring up
some war or other, in order that the people may require a leader. -- Plato
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20211209/0e8fe135/attachment.sig>