[Ffmpeg-devel] Snow segfault bug in revision 1.43

[Yartrebo]

On Tue, 2005-04-12 at 00:15 +0200, Guillaume Poirier wrote:
> Hi,
> 
> On Apr 11, 2005 10:10 PM, Guillaume Poirier <poirierg at gmail.com> wrote:
> > Hi,
> > 
> > On Apr 11, 2005 5:52 AM, Yartrebo <yartrebo at earthlink.net> wrote:
> > > On Sun, 2005-04-10 at 22:13 -0400, Yartrebo wrote:
> > > > When encoding or decoding 400x300 video, snow revisions 1.43 and onwards
> > > > cause a segmentation fault in predict_slice.
> > > >
> > > > Robert
> > > >
> > > Bug found, patch attached.
> > > Turns out that for heights that are not a multiple of the block height
> > > (16), predict_slice writes past the end of the buffer. I've fixed it.
> > 
> > Tested on AMD-64, no problem.
> 
> Well, now I'm not too sure. Your patch adds i386/snow_mmx_sse2.h which
> doesn't seem to get included on my build as the build goes though not
> matter it's in libavcodec/i386/snow_mmx_sse2.h or
> main/i386/snow_mmx_sse2.h (which is the spot where it got created by
> "patch" at first.
> 
> So I'd say patch not tested on AMD-64... :-(
> 
> Guillaume
> 
snow_mmx_sse2.h is an include file, and thus is not compiled itself. If
HAVE_MMX is defined, the file will be included and the code will be
inserted into snow.c. It's a design decision on my part.

If you want to see exactly how it got compiled in, compile snow.c with
the -E parameter, so that GCC spits out preprocessed C code. There
should be a few large hunks of assembly in add_yblock_buffered if it is
being included.

BTW, that file is part of my asm patch, which is a different patch from
the bug fix one. The bug fix patch fixes a bug from before I even
started working on snow. It's just a coincidence that both deal with
predict_slice.

Robert