[FFmpeg-devel] [PATCH] swscale: avoid overread in planar2x

[Ramiro Polla]

On Sun, Sep 12, 2010 at 7:44 AM, Michael Niedermayer <michaelni at gmx.at> wrote:
> On Sun, Sep 12, 2010 at 12:11:20PM +0200, Michael Niedermayer wrote:
>> On Sun, Sep 12, 2010 at 12:11:58AM -0300, Ramiro Polla wrote:
>> > The MMX2/3dnow code in planar2x in rgb2rgb currently reads one byte
>> > prior to the source buffer. This leads to a crash on Windows and Mac
>> > OS X when the buffer is allocated on the beginning of a page. This
>> > should also be a problem on Linux but I haven't seen many crashes
>> > because of overreading...
>>
>> the code looks buggy
>> the first pixel should be [0] not [-1] that can be achived by
>> moving the reading code of it to the end of the loop and constructing
>> the contents of the first mm4 outside the loop
>
> after a second look i see that the C code overrides the first pixel and thus
> the output should not be affected, still my suggestion though seems nicer than
> requireing all buffers to have a byte allocated before them

Thanks for the suggestion. Patch attached. No speed difference measured.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: planar2x_avoid_overread_2.diff
Type: text/x-patch
Size: 1535 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20100912/146d2848/attachment.bin>