[FFmpeg-cvslog] avformat/lxfdec: Fix multiple integer overflows related to track_size

[Michael Niedermayer]

ffmpeg | branch: master | Michael Niedermayer <michael at niedermayer.cc> | Thu Jan 14 22:08:25 2021 +0100| [7819412f4468514a2bab924291d79806a569388c] | committer: Michael Niedermayer

avformat/lxfdec: Fix multiple integer overflows related to track_size

Fixes: signed integer overflow: 538976288 * 8 cannot be represented in type 'int'
Fixes: 26910/clusterfuzz-testcase-minimized-ffmpeg_dem_LXF_fuzzer-6634030636335104

Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=7819412f4468514a2bab924291d79806a569388c
---

 libavformat/lxfdec.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/libavformat/lxfdec.c b/libavformat/lxfdec.c
index fa84ceea78..509d19fe7f 100644
--- a/libavformat/lxfdec.c
+++ b/libavformat/lxfdec.c
@@ -195,7 +195,7 @@ static int get_packet_header(AVFormatContext *s)
             return AVERROR_PATCHWELCOME;
         }
 
-        samples = track_size * 8 / st->codecpar->bits_per_coded_sample;
+        samples = track_size * 8LL / st->codecpar->bits_per_coded_sample;
 
         //use audio packet size to determine video standard
         //for NTSC we have one 8008-sample audio frame per five video frames
@@ -210,6 +210,8 @@ static int get_packet_header(AVFormatContext *s)
             avpriv_set_pts_info(s->streams[0], 64, 1, 25);
         }
 
+        if (av_popcount(channels) * (uint64_t)track_size > INT_MAX)
+            return AVERROR_INVALIDDATA;
         //TODO: warning if track mask != (1 << channels) - 1?
         ret = av_popcount(channels) * track_size;