[FFmpeg-devel] [patch] libpostproc: mmx code uses stack below %esp

[Yuriy Kaminskiy]

Hello!
While trying to catch unrelated bug (that finally was not bug, but
misconfiguration [-lavdopts fast was too fragile for broken mpeg-2 stream]),
I've run mplayer under valgrind, and got bunch of warnings:
==32414==
==32414== Invalid write of size 8
==32414==    at 0x874CF44: postProcess_MMX2 (in /path/to/mplayer)
==32414==  Address 0xbeffa9d0 is just below the stack ptr.  To suppress, use:
--workaround-gcc296-bugs=yes
I, of course, don't use gcc-2.96 ;-)
I've looked into libpostproc/postprocess_template.c, and, indeed, it uses memory
below %esp:
=== cut ===
static inline void RENAME(doVertDefFilter)(uint8_t src[], int stride, PPContext *c)
{
[...]
    __asm__ volatile(
        "pxor %%mm7, %%mm7                      \n\t"
        "lea -40(%%"REG_SP"), %%"REG_c"         \n\t" // make space for 4 8-byte
vars
        "and "ALIGN_MASK", %%"REG_c"            \n\t" // align
...
}
[...]
static inline void RENAME(dering)(uint8_t src[], int stride, PPContext *c)
[...same...]
static av_always_inline void RENAME(do_a_deblock)(uint8_t *src, int step, int
stride, PPContext *c){
[...same...]
=== cut ===
Not sure if this *must* be fixed, but it feels unsafe, so...
Patch attached; doVertDefFilter and do_a_deblock changes should not affect
speed, not sure about dering one.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: postproc-invalid-stack-3.patch
Type: text/x-diff
Size: 10365 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20100129/785f33fb/attachment.patch>