[FFmpeg-devel] [PATCH 01/11] avcodec/adpcm: Fix integer overflow in intermediate in ADPCM_XMD

[Michael Niedermayer]

Fixes: runtime error: signed integer overflow: 2140143616 + 254665816 cannot be represented in type 'int'
Fixes: 45982/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_ADPCM_XMD_fuzzer-6690181676924928

As a sideeffect this simplifies the equation, the high bits are different after this but only
the low 16bits are stored and used in later steps.
The change is untested as there are no fate testcases, no sample files on the server, no links on
the mailing list and no reports on trac referencing this format that i could find.

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
---
 libavcodec/adpcm.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavcodec/adpcm.c b/libavcodec/adpcm.c
index 451696932d1..d8f334cf5a0 100644
--- a/libavcodec/adpcm.c
+++ b/libavcodec/adpcm.c
@@ -1579,11 +1579,11 @@ static int adpcm_decode_frame(AVCodecContext *avctx, AVFrame *frame,
                     nibble[0] = sign_extend(byte & 15, 4);
                     nibble[1] = sign_extend(byte >> 4, 4);
 
-                    out[2+n*2] = (nibble[0]*(scale<<14) + (history[0]*29336) - (history[1]*13136)) >> 14;
+                    out[2+n*2  ] = nibble[0]*scale + ((history[0]*3667 - history[1]*1642) >> 11);
                     history[1] = history[0];
                     history[0] = out[2+n*2];
 
-                    out[2+n*2+1] = (nibble[1]*(scale<<14) + (history[0]*29336) - (history[1]*13136)) >> 14;
+                    out[2+n*2+1] = nibble[1]*scale + ((history[0]*3667 - history[1]*1642) >> 11);
                     history[1] = history[0];
                     history[0] = out[2+n*2+1];
                 }
-- 
2.17.1