[FFmpeg-devel] [PATCH 2/2] avcodec/pixlet: Fixes: undefined shift in av_mod_uintp2()

[Paul B Mahol]

On 8/19/17, Michael Niedermayer <michael at niedermayer.cc> wrote:
> On Fri, Aug 18, 2017 at 06:21:56PM +0200, Paul B Mahol wrote:
>> On 8/18/17, Michael Niedermayer <michael at niedermayer.cc> wrote:
>> > Fixes: runtime error: shift exponent 4294967289 is too large for 32-bit
>> > type
>> > 'int'
>> > Fixes: 3030/clusterfuzz-testcase-minimized-4649809254285312
>> >
>> > Found-by: continuous fuzzing process
>> > https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
>> > Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
>> > ---
>> >  libavcodec/pixlet.c | 2 +-
>> >  1 file changed, 1 insertion(+), 1 deletion(-)
>> >
>> > diff --git a/libavcodec/pixlet.c b/libavcodec/pixlet.c
>> > index 088226bdda..a9cfe085c9 100644
>> > --- a/libavcodec/pixlet.c
>> > +++ b/libavcodec/pixlet.c
>> > @@ -262,7 +262,7 @@ static int read_high_coeffs(AVCodecContext *avctx,
>> > uint8_t *src, int16_t *dst, i
>> >
>> >          flag = 0;
>> >
>> > -        if (state * 4ULL > 0xFF || i >= size)
>> > +        if ((uint64_t)state > 0xFF / 4 || i >= size)
>>
>> This is not exact same behaviour.
>
> no, it differs for a small number of cases that have one of the 2
> most significant bits (of the 64 bits) set in state.
> I think this was a bug i introduced in a previous commit.
>
> If above change is not the correct thing to do, please elaborate.
> state is passed into ff_clz() later which has no deterministic
> behavior when these MSBs are set as ff_clz() uses unsigned and state
> is int64_t and unsigned can be 32 or 64bit.

Only 32 bits matters for ff_clz() anyway.

> So this really just change cases that do not work equally
> across platforms
>
> looking at this now, i suspect independant of this patch
> the 32 and 24 in
> pfx = ((state + 8) >> 5) + (state ? ff_clz(state): 32) - 24;
> should be CHAR_BIT * sizeof(unsiged) or something similar
> or something else should be changed
>
> [...]
> --
> Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
>
> Take away the freedom of one citizen and you will be jailed, take away
> the freedom of all citizens and you will be congratulated by your peers
> in Parliament.
>