[FFmpeg-devel] [PATCH 4/5] avcodec/apv_dsp: Avoid UB overflow in dequant

[Michael Niedermayer]

Fixes: signed integer overflow: 33632416 * 64 cannot be represented in type 'int'
Fixes: 421817631/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_APV_fuzzer-4957386534354944

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
---
 libavcodec/apv_dsp.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavcodec/apv_dsp.c b/libavcodec/apv_dsp.c
index 07bb34ec0bf..8fbabcf63db 100644
--- a/libavcodec/apv_dsp.c
+++ b/libavcodec/apv_dsp.c
@@ -58,8 +58,8 @@ static void apv_decode_transquant_c(void *output,
 
         for (int y = 0; y < 8; y++) {
             for (int x = 0; x < 8; x++) {
-                int coeff = (input[y][x] * qmatrix[y][x] * (1 << qp_shift) +
-                             (1 << (bd_shift - 1))) >> bd_shift;
+                int coeff = ((int)(input[y][x] * qmatrix[y][x] * (1U << qp_shift) +
+                             (1 << (bd_shift - 1)))) >> bd_shift;
 
                 scaled_coeff[y][x] =
                     av_clip(coeff, APV_MIN_TRANS_COEFF,
-- 
2.49.0