[FFmpeg-cvslog] avformat/rpl: Use 64bit for total_audio_size and check it

[Michael Niedermayer]

ffmpeg | branch: release/2.8 | Michael Niedermayer <michael at niedermayer.cc> | Tue Mar 26 00:36:40 2024 +0100| [f0d1868cda102e9a13e9789acc81278c5f936949] | committer: Michael Niedermayer

avformat/rpl: Use 64bit for total_audio_size and check it

Fixes: 62276/clusterfuzz-testcase-minimized-ffmpeg_dem_RPL_fuzzer-4677434693517312
Fixes: signed integer overflow: 5555555555555555556 * 8 cannot be represented in type 'long long'

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit 878625812f164fbb733f442965235656d9eaccc8)
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=f0d1868cda102e9a13e9789acc81278c5f936949
---

 libavformat/rpl.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/libavformat/rpl.c b/libavformat/rpl.c
index d2f9d6ec13..6f591f128b 100644
--- a/libavformat/rpl.c
+++ b/libavformat/rpl.c
@@ -117,7 +117,7 @@ static int rpl_read_header(AVFormatContext *s)
     AVIOContext *pb = s->pb;
     RPLContext *rpl = s->priv_data;
     AVStream *vst = NULL, *ast = NULL;
-    int total_audio_size;
+    int64_t total_audio_size;
     int error = 0;
 
     uint32_t i;
@@ -270,6 +270,8 @@ static int rpl_read_header(AVFormatContext *s)
         if (ast)
             av_add_index_entry(ast, offset + video_size, total_audio_size,
                                audio_size, audio_size * 8, 0);
+        if (total_audio_size/8 + (uint64_t)audio_size >= INT64_MAX/8)
+            return AVERROR_INVALIDDATA;
         total_audio_size += audio_size * 8;
     }