[FFmpeg-devel] [PATCH] libopenjpegenc: fix out-of-bounds reads when filling the edges

[Andreas Cadhalpun]

On 14.10.2016 00:00, Hendrik Leppkes wrote:
> On Thu, Oct 13, 2016 at 10:25 PM, Andreas Cadhalpun
> <andreas.cadhalpun at googlemail.com> wrote:
>> If x is 0, 'x - 1' is in the previous line, or worse outside the buffer
>> for the first line.
>>
>> If y is 0, 'x - image->comps[compno].w' is outside the buffer.
>>
> 
> I'm slightly puzzled, as you say, these are for edge handling, edges
> in this case are from the image width to buffer width, and image
> height to buffer height, respectively
> So for x or y to be zero, we would need an image thats zero width, or
> zero height, so the edge starts at zero?
> 
> How does that happen, and wouldn't it be much simpler to catch that
> case earlier in the chain and simply error out? A image with either
> zero width or zero height surely is not something you can encode
> either way.

The avctx->width/avctx->height is not zero, but libopenjpeg_copy_unpacked8
does:
        width  = avctx->width / image->comps[compno].dx;
        height = avctx->height / image->comps[compno].dy;

So if e.g. avctx->height is 1 and image->comps[compno].dy is 2, height
becomes 0. I'm not sure if that's invalid.

I think the other functions are currently not affected by this, but I
added the check anyway for robustness. (The cast to int is needed in
any case.)

Best regards,
Andreas