[FFmpeg-devel] [PATCH] Prevent integer overflow in vf_psnr.c.

[Neil Birkbeck]

The 32-bit integer accumulator in MSE computation can overflow for 8-bit frame data.  
(e.g., for 1080p white frame compared to a black frame can give sum of 255*255*1080*1920 > 2^32).

Signed-off-by: Neil Birkbeck <neil.birkbeck at gmail.com>
---
 libavfilter/vf_psnr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavfilter/vf_psnr.c b/libavfilter/vf_psnr.c
index 3744579..1bcdd7d 100644
--- a/libavfilter/vf_psnr.c
+++ b/libavfilter/vf_psnr.c
@@ -91,7 +91,7 @@ void compute_images_mse(PSNRContext *s,
         const uint8_t *ref_line = ref_data[c];
         const int ref_linesize = ref_linesizes[c];
         const int main_linesize = main_linesizes[c];
-        int m = 0;
+        uint64_t m = 0;
 
         for (i = 0; i < outh; i++) {
             for (j = 0; j < outw; j++)
-- 
1.8.4