[FFmpeg-devel] [PATCH] avcodec/vp9block: fix runtime error: signed integer overflow: 196675 * 20670 cannot be represented in type 'int'

[Ronald S. Bultje]

Hi,

On Sat, May 20, 2017 at 8:12 PM, Michael Niedermayer <michael at niedermayer.cc
> wrote:

> Fixes: 1710/clusterfuzz-testcase-minimized-4837032931098624
>
> Found-by: continuous fuzzing process https://github.com/google/oss-
> fuzz/tree/master/projects/ffmpeg
> Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> ---
>  libavcodec/vp9block.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/libavcodec/vp9block.c b/libavcodec/vp9block.c
> index ae2f0e4c6f..a16ccdccdb 100644
> --- a/libavcodec/vp9block.c
> +++ b/libavcodec/vp9block.c
> @@ -915,9 +915,9 @@ skip_eob:
>          if (!--band_left)
>              band_left = band_counts[++band];
>          if (is_tx32x32)
> -            STORE_COEF(coef, rc, ((vp8_rac_get(c) ? -val : val) *
> qmul[!!i]) / 2);
> +            STORE_COEF(coef, rc, (int)((vp8_rac_get(c) ? -val : val) *
> (unsigned)qmul[!!i]) / 2);
>          else
> -            STORE_COEF(coef, rc, (vp8_rac_get(c) ? -val : val) *
> qmul[!!i]);
> +            STORE_COEF(coef, rc, (vp8_rac_get(c) ? -val : val) *
> (unsigned)qmul[!!i]);
>          nnz = (1 + cache[nb[i][0]] + cache[nb[i][1]]) >> 1;
>          tp = p[band][nnz];
>      } while (++i < n_coeffs);
> --
> 2.13.0


Since this is the only use of qmul[], why don't you make the array unsigned
instead? That saves a cast.

Ronald