[FFmpeg-devel] [PATCH] Correct MS RLE decoding

[Kostya]

On Sat, May 16, 2009 at 03:15:02AM +0200, Michael Niedermayer wrote:
> On Tue, May 12, 2009 at 08:08:42AM +0300, Kostya wrote:
> > Looks like M$ RLE and its friends (TSCC, AASC) code an additional line
> > when height is odd.
> > 
> > The patch attached was intended to fix off-by-one error for "line"
> > variable which fixes issue 1069 but during that this side effect was
> > discovered and an additional care is needed.
> 
> >  msrledec.c |    6 +++---
> >  1 file changed, 3 insertions(+), 3 deletions(-)
> > 4745e36d382ba1e740b12e0e0c020e31615cb415  rle.patch
> > Index: libavcodec/msrledec.c
> > ===================================================================
> > --- libavcodec/msrledec.c	(revision 18797)
> > +++ libavcodec/msrledec.c	(working copy)
> > @@ -134,19 +134,19 @@
> >  {
> >      uint8_t *output, *output_end;
> >      const uint8_t* src = data;
> > -    int p1, p2, line=avctx->height, pos=0, i;
> > +    int p1, p2, line=avctx->height-!(avctx->height&1), pos=0, i;
> >      uint16_t av_uninit(pix16);
> >      uint32_t av_uninit(pix32);
> >  
> >      output = pic->data[0] + (avctx->height - 1) * pic->linesize[0];
> > -    output_end = pic->data[0] + (avctx->height) * pic->linesize[0];
> > +    output_end = pic->data[0] + (avctx->height + (avctx->height & 1)) * pic->linesize[0];
> >      while(src < data + srcsize) {
> >          p1 = *src++;
> >          if(p1 == 0) { //Escape code
> >              p2 = *src++;
> >              if(p2 == 0) { //End-of-line
> >                  output = pic->data[0] + (--line) * pic->linesize[0];
> > -                if (line < 0){
> > +                if (line < 0 && src[0] && src[1] != 1){
> >                      av_log(avctx, AV_LOG_ERROR, "Next line is beyond picture bounds\n");
> >                      return -1;
> >                  }
> 
> your code looks exploitable
 
I've committed in r18852 a fix for the situation when decoder to tried
writing past picture when first line of coded frame contained skip.

As for that situation - I'll investigate a bit more.
 
> -- 
> Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB