[FFmpeg-devel] [PATCH] swscale: avoid overread in planar2x

[Michael Niedermayer]

On Sun, Sep 12, 2010 at 10:00:07AM -0300, Ramiro Polla wrote:
> On Sun, Sep 12, 2010 at 7:44 AM, Michael Niedermayer <michaelni at gmx.at> wrote:
> > On Sun, Sep 12, 2010 at 12:11:20PM +0200, Michael Niedermayer wrote:
> >> On Sun, Sep 12, 2010 at 12:11:58AM -0300, Ramiro Polla wrote:
> >> > The MMX2/3dnow code in planar2x in rgb2rgb currently reads one byte
> >> > prior to the source buffer. This leads to a crash on Windows and Mac
> >> > OS X when the buffer is allocated on the beginning of a page. This
> >> > should also be a problem on Linux but I haven't seen many crashes
> >> > because of overreading...
> >>
> >> the code looks buggy
> >> the first pixel should be [0] not [-1] that can be achived by
> >> moving the reading code of it to the end of the loop and constructing
> >> the contents of the first mm4 outside the loop
> >
> > after a second look i see that the C code overrides the first pixel and thus
> > the output should not be affected, still my suggestion though seems nicer than
> > requireing all buffers to have a byte allocated before them
> 
> Thanks for the suggestion. Patch attached. No speed difference measured.

>  rgb2rgb_template.c |    4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 9c5b0f0bc55c99bcfbda486bff43d26c34cb96f5  planar2x_avoid_overread_2.diff
> Index: rgb2rgb_template.c
> ===================================================================
> --- rgb2rgb_template.c	(revision 32166)
> +++ rgb2rgb_template.c	(working copy)
> @@ -1773,12 +1773,13 @@ static inline void RENAME(planar2x)(const uint8_t
>          const x86_reg mmxSize= srcWidth&~15;
>          __asm__ volatile(
>              "mov           %4, %%"REG_a"            \n\t"
> +            "movq         (%0, %%"REG_a"), %%mm4    \n\t"
> +            "psllq                     $8, %%mm4    \n\t"

the first value is then 0 which would lead to a darker border pixel


[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

No snowflake in an avalanche ever feels responsible. -- Voltaire
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20100912/627e7e77/attachment.pgp>