[FFmpeg-devel] stsz overflow

[Frank Barchard]

On Mon, Aug 24, 2009 at 4:08 PM, Alex Converse <alex.converse at gmail.com>wrote:

>
> >     num_bytes = (entries*field_size+4)>>3;
>
> The intermediate product here is the part that overflows. A final
> num_bytes calculated with appropriate intermediate precision should
> fit in in an unsigned 32-bit integer. Why not just fix that rather
> than reduce the number of entries supported?


Alex,
Ya, you're right.  I see other code that uses uint64_t to calculate sizes
for malloc, and then it would have a chance of success.
The code that follows the failed malloc returns an error code, which is
good.

    num_bytes = (unsigned int)(((uint64_t)entries*field_size+7)>>3);
    buf = av_malloc(num_bytes+FF_INPUT_BUFFER_PADDING_SIZE);    if
(!buf) {        av_freep(&sc->sample_sizes);        return
AVERROR(ENOMEM);    }

updated patch here:


Index: libavformat/mov.c
===================================================================
--- libavformat/mov.c   (revision 19695)
+++ libavformat/mov.c   (working copy)
@@ -1262,7 +1262,7 @@
     if (!sc->sample_sizes)
         return AVERROR(ENOMEM);

-    num_bytes = (entries*field_size+4)>>3;
+    num_bytes = (unsigned int)(((uint64_t)entries*field_size+4)>>3);

     buf = av_malloc(num_bytes+FF_INPUT_BUFFER_PADDING_SIZE);
     if (!buf) {