[FFmpeg-devel] [PATCH] lsws: prevent overflow in sws_init_context()

[Stefano Sabatini]

In the loop:
    for (i=0; i<dstH; i++) {
        int chrI= i*c->chrDstH / dstH;

when i*c->chrDstH > INT_MAX this leads to an integer overflow, which
results in a negative value for chrI and in out-of-buffer reads. The
overflow is avoided by forcing int64_t arithmetic by casting i to
int64_t.

Fix crash, and trac issue #72.
---
 libswscale/utils.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)

diff --git a/libswscale/utils.c b/libswscale/utils.c
index a343bf2..35bf143 100644
--- a/libswscale/utils.c
+++ b/libswscale/utils.c
@@ -996,7 +996,7 @@ int sws_init_context(SwsContext *c, SwsFilter *srcFilter, SwsFilter *dstFilter)
     c->vLumBufSize= c->vLumFilterSize;
     c->vChrBufSize= c->vChrFilterSize;
     for (i=0; i<dstH; i++) {
-        int chrI= i*c->chrDstH / dstH;
+        int chrI= (int64_t)i*c->chrDstH / dstH;
         int nextSlice= FFMAX(c->vLumFilterPos[i   ] + c->vLumFilterSize - 1,
                            ((c->vChrFilterPos[chrI] + c->vChrFilterSize - 1)<<c->chrSrcVSubSample));
 
-- 
1.7.2.3