[FFmpeg-cvslog] avcodec/h264: Undefined behavior ( left shift of 12852653 by 8 places cannot be represented in type 'int')

[Michael Niedermayer]

ffmpeg | branch: master | Michael Niedermayer <michaelni at gmx.at> | Wed Oct  8 04:23:03 2014 +0200| [f3296b945464b41aa067949b24dfcfeb0db9d875] | committer: Michael Niedermayer

avcodec/h264: Undefined behavior (left shift of 12852653 by 8 places cannot be represented in type 'int')

Fixes: asan_heap-oob_84f75d_8_asan_heap-oob_a2a00a_341_mbc.ts

Found-by: Mateusz "j00ru" Jurczyk and Gynvael Coldwind
Signed-off-by: Michael Niedermayer <michaelni at gmx.at>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=f3296b945464b41aa067949b24dfcfeb0db9d875
---

 libavcodec/h264.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavcodec/h264.c b/libavcodec/h264.c
index c4f4b05..0405779 100644
--- a/libavcodec/h264.c
+++ b/libavcodec/h264.c
@@ -1358,7 +1358,7 @@ static int get_avc_nalsize(H264Context *h, const uint8_t *buf,
         return -1;
 
     for (i = 0; i < h->nal_length_size; i++)
-        nalsize = (nalsize << 8) | buf[(*buf_index)++];
+        nalsize = ((unsigned)nalsize << 8) | buf[(*buf_index)++];
     if (nalsize <= 0 || nalsize > buf_size - *buf_index) {
         av_log(h->avctx, AV_LOG_ERROR,
                "AVC: nal size %d\n", nalsize);