[FFmpeg-devel] [PATCH] avcodec/agm: add support for higher compression

[James Almer]

On 4/8/2019 12:42 PM, Paul B Mahol wrote:
>>> +static int decode_huffman2(AVCodecContext *avctx, int header, int size)
>>> +{
>>> +    AGMContext *s = avctx->priv_data;
>>> +    GetBitContext *gb = &s->gb;
>>> +    uint8_t lens[256];
>>> +    uint32_t output_size;
>>> +    int ret, x, len;
>>> +
>>> +    if ((ret = init_get_bits8(gb, s->gbyte.buffer,
>>> +                              bytestream2_get_bytes_left(&s->gbyte))) <
>>> 0)
>>> +        return ret;
>>> +
>>> +    output_size = get_bits_long(gb, 32);
>>> +
>>> +    av_fast_padded_malloc(&s->output, &s->output_size,
>>> +                          output_size * sizeof(*s->output));
>> Several chances for overflow here.
> Yes, changed output_size to int.

No, it needs to be unsigned for av_fast_padded_malloc(). What you need
to also make unsigned is s->output_size instead.

Also, that sizeof(*s->output) seems superfluous.