[FFmpeg-cvslog] avcodec/aacpsdsp_template: Fix integer overflows in ps_decorrelate_c()

[Michael Niedermayer]

ffmpeg | branch: release/3.2 | Michael Niedermayer <michael at niedermayer.cc> | Sun Nov  5 21:20:08 2017 +0100| [4654cc8cee07b7772239eec7d263d6cd9c6a4f0d] | committer: Michael Niedermayer

avcodec/aacpsdsp_template: Fix integer overflows in ps_decorrelate_c()

Fixes: runtime error: signed integer overflow: 1939661764 - -454942263 cannot be represented in type 'int'
Fixes: 3191/clusterfuzz-testcase-minimized-5688798451073024

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit 2afe05402f05d485f0c356b04dc562f0510d317d)
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=4654cc8cee07b7772239eec7d263d6cd9c6a4f0d
---

 libavcodec/aacpsdsp_template.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/libavcodec/aacpsdsp_template.c b/libavcodec/aacpsdsp_template.c
index 3049ce8b79..0e532fcf84 100644
--- a/libavcodec/aacpsdsp_template.c
+++ b/libavcodec/aacpsdsp_template.c
@@ -129,12 +129,12 @@ static void ps_decorrelate_c(INTFLOAT (*out)[2], INTFLOAT (*delay)[2],
             INTFLOAT apd_im = in_im;
             in_re = AAC_MSUB30(link_delay_re, fractional_delay_re,
                     link_delay_im, fractional_delay_im);
-            in_re -= a_re;
+            in_re -= (UINTFLOAT)a_re;
             in_im = AAC_MADD30(link_delay_re, fractional_delay_im,
                     link_delay_im, fractional_delay_re);
-            in_im -= a_im;
-            ap_delay[m][n+5][0] = apd_re + AAC_MUL31(ag[m], in_re);
-            ap_delay[m][n+5][1] = apd_im + AAC_MUL31(ag[m], in_im);
+            in_im -= (UINTFLOAT)a_im;
+            ap_delay[m][n+5][0] = apd_re + (UINTFLOAT)AAC_MUL31(ag[m], in_re);
+            ap_delay[m][n+5][1] = apd_im + (UINTFLOAT)AAC_MUL31(ag[m], in_im);
         }
         out[n][0] = AAC_MUL16(transient_gain[n], in_re);
         out[n][1] = AAC_MUL16(transient_gain[n], in_im);