[MPlayer-users] SIGSEGV: mencoder + mpeg2video + vpass=2

[Nico Sabbi]

byavl wrote:

> Hi All.
>
> It's me again with SIGSEGV :)
>
> Troble is a function in mjpeg.c:
> static int mjpeg_decode_frame(AVCodecContext *avctx,
>                               void *data, int *data_size,
>                               uint8_t *buf, int buf_size)
>
> There is code where don't check buffer pointer position
> to leave bounds into "while{}" construction.
> Of course SIGSEGV welcome.
>
> [avl at avl argasi]$ gdb mencoder --core=core.14215
> GNU gdb Red Hat Linux (5.3post-0.20021129.18rh)
> This GDB was configured as "i386-redhat-linux-gnu"...
> Core was generated by `mencoder argasi96_raw1_25.mjpg -o 
> argasi96_raw1_25.mjpg.mpeg -vf hqdn3d=2:1:2,p'.
> Program terminated with signal 11, Segmentation fault.
>
> #0  0x081b575f in mjpeg_decode_frame (avctx=0x865fe20, data=0x865fd30, 
> data_size=0x30, buf=0x881aec0 "ЪьЪЮ",
>     buf_size=48) at mjpeg.c:1837
> 1837    mjpeg.c: No such file or directory.
>         in mjpeg.c
>
> mjpeg.c
> line 1822:
>
>    /* unescape buffer of SOS */
>    if (start_code == SOS)
>    {
>        uint8_t *src = buf_ptr;
>        uint8_t *dst = s->buffer;
>
>        while (src<buf_end)
>        {
>            uint8_t x = *(src++);
>
>            *(dst++) = x;
>            if (x == 0xff)
>            {
>                while(*src == 0xff) src++;
>
> line 1837:    x = *(src++); <-- SIGSEGV here,but causer is prev. while.
>                if (x >= 0xd0 && x <= 0xd7)
>                    *(dst++) = x;
>                else if (x)
>                    break;
>            }
>        }
>        init_get_bits(&s->gb, s->buffer, (dst - s->buffer)*8);
>
>        dprintf("escaping removed %d bytes\n",
>            (buf_end - buf_ptr) - (dst - s->buffer));
>    }
>    else
>        init_get_bits(&s->gb, buf_ptr, (buf_end - buf_ptr)*8);
>
>
>
> ===
> Aleksey Lekhtin
> byavl at papillon.ru   ICQ: 15146863, AIM: byavl
>

bugfixes for libavcodec/libavformat should be posted to 
ffmpeg-devel at lists.sourceforge.net