[FFmpeg-cvslog] rtpenc: fix overflow checking in avc_mp4_find_startcode()

[Xi Wang]

ffmpeg | branch: release/0.11 | Xi Wang <xi.wang at gmail.com> | Tue Jan 22 20:58:07 2013 -0500| [a31be9dd065bc294c214dfaad9abd08fe3919569] | committer: Michael Niedermayer

rtpenc: fix overflow checking in avc_mp4_find_startcode()

The check `start + res < start' is broken since pointer overflow is
undefined behavior in C.  Many compilers such as gcc/clang optimize
away this check.

Use `res > end - start' instead.  Also change `res' to unsigned int
to avoid signed left-shift overflow.

Signed-off-by: Xi Wang <xi.wang at gmail.com>
Signed-off-by: Michael Niedermayer <michaelni at gmx.at>
(cherry picked from commit 2f014567cfd63e58156f60666f1a61ba147276ab)

Signed-off-by: Michael Niedermayer <michaelni at gmx.at>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=a31be9dd065bc294c214dfaad9abd08fe3919569
---

 libavformat/rtpenc_h264.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/libavformat/rtpenc_h264.c b/libavformat/rtpenc_h264.c
index 86930bb..5447edc 100644
--- a/libavformat/rtpenc_h264.c
+++ b/libavformat/rtpenc_h264.c
@@ -31,14 +31,14 @@
 
 static const uint8_t *avc_mp4_find_startcode(const uint8_t *start, const uint8_t *end, int nal_length_size)
 {
-    int res = 0;
+    unsigned int res = 0;
 
     if (end - start < nal_length_size)
         return NULL;
     while (nal_length_size--)
         res = (res << 8) | *start++;
 
-    if (start + res > end || res < 0 || start + res < start)
+    if (res > end - start)
         return NULL;
 
     return start + res;