[FFmpeg-cvslog] avutil/imgutils: av_image_check_size2() ensure width and height fit in 32bit

[Michael Niedermayer]

ffmpeg | branch: release/2.8 | Michael Niedermayer <michael at niedermayer.cc> | Wed Jul 10 17:49:56 2024 +0200| [4c1908393bf2d8434be568cea92a76892b60f9a3] | committer: Michael Niedermayer

avutil/imgutils: av_image_check_size2() ensure width and height fit in 32bit

width and height > 32bit is not supported and its easier to check in a central place

Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
(cherry picked from commit ba63e329572b74207045fd82c93fcc0fa0479bc4)
Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=4c1908393bf2d8434be568cea92a76892b60f9a3
---

 libavutil/imgutils.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavutil/imgutils.c b/libavutil/imgutils.c
index b727b6b5f5..8e3c19333e 100644
--- a/libavutil/imgutils.c
+++ b/libavutil/imgutils.c
@@ -246,7 +246,7 @@ int av_image_check_size2(unsigned int w, unsigned int h, int64_t max_pixels, enu
         stride = 8LL*w;
     stride += 128*8;
 
-    if ((int)w<=0 || (int)h<=0 || stride >= INT_MAX || stride*(uint64_t)(h+128) >= INT_MAX) {
+    if (w==0 || h==0 || w > INT32_MAX || h > INT32_MAX || stride >= INT_MAX || stride*(h + 128ULL) >= INT_MAX) {
         av_log(&imgutils, AV_LOG_ERROR, "Picture size %ux%u is invalid\n", w, h);
         return AVERROR(EINVAL);
     }