[FFmpeg-cvslog] avcodec/vmnc: Check that rectangles are within the picture

[Michael Niedermayer]

ffmpeg | branch: release/0.5 | Michael Niedermayer <michaelni at gmx.at> | Mon Jan 20 18:08:18 2014 +0100| [43aae004557294ffe3487bb4b194c3eb72beb27b] | committer: Michael Niedermayer

avcodec/vmnc: Check  that rectangles are within the picture

Prevents out of array accesses with CODEC_FLAG_EMU_EDGE

Signed-off-by: Michael Niedermayer <michaelni at gmx.at>
(cherry picked from commit 6ba02602aa7fc7d38db582e75b8b093fb3c1608d)

Conflicts:

	libavcodec/vmnc.c

Signed-off-by: Michael Niedermayer <michaelni at gmx.at>
(cherry picked from commit 7c17207ab9acfaa934e8feb8fba90765c9d0b989)

Signed-off-by: Michael Niedermayer <michaelni at gmx.at>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=43aae004557294ffe3487bb4b194c3eb72beb27b
---

 libavcodec/vmnc.c |    5 +++++
 1 file changed, 5 insertions(+)

diff --git a/libavcodec/vmnc.c b/libavcodec/vmnc.c
index c7efaf8..aafd89c 100644
--- a/libavcodec/vmnc.c
+++ b/libavcodec/vmnc.c
@@ -275,6 +275,11 @@ static int decode_hextile(VmncContext *c, uint8_t* dst, const uint8_t* src, int
                     }
                     xy = *src++;
                     wh = *src++;
+                    if (   (xy >> 4) + (wh >> 4) + 1 > w - i
+                        || (xy & 0xF) + (wh & 0xF)+1 > h - j) {
+                        av_log(c->avctx, AV_LOG_ERROR, "Rectangle outside picture\n");
+                        return AVERROR_INVALIDDATA;
+                    }
                     paint_rect(dst2, xy >> 4, xy & 0xF, (wh>>4)+1, (wh & 0xF)+1, fg, bpp, stride);
                 }
             }