[FFmpeg-devel] [PATCH 2/2] avcodec/sanm: avoid using k in left pxoff check

[Michael Niedermayer]

On Tue, Jun 03, 2025 at 12:30:40PM +0200, Manuel Lauss wrote:
> Servus Michael,
> 
> On Sat, May 31, 2025 at 12:51 AM Michael Niedermayer
> <michael at niedermayer.cc> wrote:
> 
> > > > >              /* smooth top and left block borders with neighbours */
> > > > > -            if (((pxoff - p + k) < 0) || ((pxoff - p + k) >= maxpxo)
> > > > > +            if (((pxoff - p + 0) < 0) || ((pxoff - p + k) >= maxpxo)
> > > > >                  || ((pxoff + 3 * p) < 0) || ((pxoff + 3 * p) >= maxpxo)
> > > > >                  || (i == 0) || (j == 0))
> > > > >                  continue;
> > >
> > > Oops, yes, that change is correct.
> >
> > will apply
> >
> >
> > > I think you should just rip the whole block-smoothing block out
> > > entirely: it creates a "sawtooth" pattern
> > > which is more annoying than the "blockiness" it tries to soften.
> >
> > probably, but i just wanted to fix the out of array access,
> > ill leave improvment beyond that to others
> 
> Is it OK if I just remove this block entirely?  I'd commit that along
> with the other 2 sanm patches I have.

i think you dont understand the bug my change is fixing

Its an out of array access in:
                for (k = 0; k < 4; k++)
                    *(dst + pxoff + k) = ((*(dst + pxoff + k) + *(dst + pxoff - p + k)) >> 1) & 0x7f;

The hunk:
            /* smooth top and left block borders with neighbours */
            if (((pxoff - p + k) < 0) || ((pxoff - p + k) >= maxpxo)
                || ((pxoff + 3 * p) < 0) || ((pxoff + 3 * p) >= maxpxo)
                || (i == 0) || (j == 0))
                continue;

skips this code. If the hunk is removed and nothing else is changed
its MORE buggy as there will be more out of array accesses


thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

For a strong democracy, genuine criticism is necessary, allegations benefit
noone, they just cause unnecessary conflicts. - Narendra Modi
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20250604/ead168fc/attachment.sig>