[FFmpeg-devel] [PATCH] Missing emms_c() calls causing weird Windows crashes with deshake

[Ray Simard]

Before I go into this, I'd like to ask if anyone else has had this
problem: ffmpeg can't run in a debugger in Windows. gdb crashes (gdb
itself, not ffmpeg) before it even gets to main(). Both Cygwin and Mingw
gdb's do the same thing. Even WinDbg crashes in the same place.  There's
no problem in Linux. I suspect some peculiarity in my system, but maybe
someone has a lead on this. In any case, I'm pretty sure that has no
bearing on this; it just made hunting this down rather tricky.


The problem I was having was an ffmpeg segfault whenever the deshake
filter was used, but only in Windows.  In Linux, there was no problem.
To skip over a lot of trial and error, I was able to establish the
following:

1. All three versions, Linux, Cygwin and Mingw, were compiled from the
same source (N-35450-g7071385 at the time).  For test I compiled with
these options:

--prefix=/usr/local_XXXX --enable-debug=3 --disable-stripping \
--disable-optimizations

where XXXX is something I used to keep track of the various experiments
I was doing.

Linux worked as advertised.  Both Windows flavors died as soon as
transcoding began.

The crash was traced to this part of transform.c:

                case FILL_MIRROR:
                    y_s = (y_s < 0) ? -y_s : (y_s >= height) ?\
 (height + height - y_s) : y_s;
                    x_s = (x_s < 0) ? -x_s : (x_s >= width) ? \
 (width + width - x_s) : x_s;
   KABOOM >>>>      def = src[(int)y_s * src_stride + (int)x_s];
            }

The floating-point values x_s and y_s (and lots of others) were NaN and
the subscript it calculated was a huge negative number.

The bad numbers were traced back to vf_deshake.c:

    if (deshake->search == EXHAUSTIVE) {
        // Compare every possible position - this is sloooow!
        for (y = -deshake->ry; y <= deshake->ry; y++) {
            for (x = -deshake->rx; x <= deshake->rx; x++) {
                diff = CMP(cx - x, cy - y);
                if (diff < smallest) {
                    smallest = diff;
                    mv->x = x;
                    mv->y = y;
                }
            }

The garbage values (specifically 0xffff800000000000, which is a
floating-point NaN) were being assigned to one or the other of the
members of mv.

The CMP macro calls assembly-language routines that use the MMX
registers, and the call to emms_c() required before floating-point
operations thereafter are possible wasn't being done.  Why that should
bother the Windows versions and not Linux on the same machine, or why
this hasn't cropped up elsewhere, is a mystery to me.  The attached
patch has fixed the problem.  FATE is OK with it.

(This is my first contribution of this sort here, so please be merciful
if I haven't quite done it right.)

HTH,
Ray Simard
rhs.ffmpeg at sylvan-glade.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: emms-after-sad.patch
Type: text/x-patch
Size: 2043 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20120106/5803dcd1/attachment.bin>