[FFmpeg-cvslog] avformat/rmdec: Check old_format len for overflow

[Michael Niedermayer]

ffmpeg | branch: release/4.4 | Michael Niedermayer <michael at niedermayer.cc> | Mon Apr 26 22:35:37 2021 +0200| [ee34b6549f76f5ad0719b48debffd7e1c292701e] | committer: Michael Niedermayer

avformat/rmdec: Check old_format len for overflow

Maybe such large values could be disallowed earlier and closer to where
they are set.

Fixes: signed integer overflow: 538976288 * 8224 cannot be represented in type 'int'
Fixes: 29102/clusterfuzz-testcase-minimized-ffmpeg_dem_RM_fuzzer-6704350354341888

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit 06d174e289eb185f03a34a738965f0042f39c038)
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=ee34b6549f76f5ad0719b48debffd7e1c292701e
---

 libavformat/rmdec.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/libavformat/rmdec.c b/libavformat/rmdec.c
index fc3bff4859..36765b4763 100644
--- a/libavformat/rmdec.c
+++ b/libavformat/rmdec.c
@@ -1012,8 +1012,8 @@ static int rm_read_packet(AVFormatContext *s, AVPacket *pkt)
 {
     RMDemuxContext *rm = s->priv_data;
     AVStream *st = NULL; // init to silence compiler warning
-    int i, len, res, seq = 1;
-    int64_t timestamp, pos;
+    int i, res, seq = 1;
+    int64_t timestamp, pos, len;
     int flags;
 
     for (;;) {
@@ -1032,7 +1032,9 @@ static int rm_read_packet(AVFormatContext *s, AVPacket *pkt)
                 ast = st->priv_data;
                 timestamp = AV_NOPTS_VALUE;
                 len = !ast->audio_framesize ? RAW_PACKET_SIZE :
-                    ast->coded_framesize * ast->sub_packet_h / 2;
+                    ast->coded_framesize * (int64_t)ast->sub_packet_h / 2;
+                if (len > INT_MAX)
+                    return AVERROR_INVALIDDATA;
                 flags = (seq++ == 1) ? 2 : 0;
                 pos = avio_tell(s->pb);
             } else {