[Mplayer-advusers] [BUG] -vf scale crashes in asm

[Jan Knutar]

Probably related to recent AMD64 patches...

Originally discovered by a user on #mplayer wishing to remain
anonymous due to pyrophobia. Wasn't able to reproduce reported
image corruption, -vf scale just crashes for me on my P3.


$ gdb ./mplayer
GNU gdb Red Hat Linux (6.0post-0.20040223.19rh)
Copyright 2004 Free Software Foundation, Inc.
[jan at polaris main]$ gdb ./mplayer
GNU gdb Red Hat Linux (6.0post-0.20040223.19rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".

(gdb) set args -v -vf scale=512:384 ~/overthehills.avi
(gdb) run
Starting program: /extra/mplayer2/main/mplayer -v -vf scale=512:384 ~/overthehills.avi
[Thread debugging using libthread_db enabled]
[New Thread -151139808 (LWP 12358)]
Using GNU internationalization
Original domain: messages
Original dirname: /usr/share/locale
Current domain: mplayer
Current dirname: /usr/local/share/locale

MPlayer dev-CVS-041021-17:02-2.95.3 (C) 2000-2004 MPlayer Team

CPU: Intel Celeron 2/Pentium III Coppermine,Geyserville (Family: 6, Stepping: 3)
Detected cache-line size is 32 bytes
CPUflags:  MMX: 1 MMX2: 1 3DNow: 0 3DNow2: 0 SSE: 1 SSE2: 0
Compiled for x86 CPU with extensions: MMX MMX2 SSE

Reading config file /usr/local/etc/mplayer/mplayer.conf: No such file or directory
Reading config file /home/jan/.mplayer/config
Reading /home/jan/.mplayer/codecs.conf: Can't open '/home/jan/.mplayer/codecs.conf': No such file or directory
Reading /usr/local/etc/mplayer/codecs.conf: Can't open '/usr/local/etc/mplayer/codecs.conf': No such file or directory
Using built-in default codecs.conf.
CommandLine: '-v' '-vf' 'scale=512:384' '/home/jan/overthehills.avi'
init_freetype
get_path('font/font.desc') -> '/home/jan/.mplayer/font/font.desc'
Font /home/jan/.mplayer/font/font.desc loaded successfully! (206 chars)
Using MMX (with tiny bit MMX2) Optimized OnScreenDisplay
Using Linux hardware RTC timing (1024Hz).
get_path('input.conf') -> '/home/jan/.mplayer/input.conf'
Parsing input config file /home/jan/.mplayer/input.conf
Input config file /home/jan/.mplayer/input.conf parsed: 52 binds
get_path('overthehills.avi.conf') -> '/home/jan/.mplayer/overthehills.avi.conf'

Playing /home/jan/overthehills.avi.
[file] File size is 19492348 bytes
STREAM: [file] /home/jan/overthehills.avi
STREAM: Description: File
STREAM: Author: Albeu
STREAM: Comment: based on the code from ??? (probably Arpi)
AVI file format detected.
list_end=0x146
======= AVI Header =======
us/frame: 40000  (fps=25.000)
max bytes/sec: 0
padding: 0
MainAVIHeader.dwFlags: (2320) HAS_INDEX IS_INTERLEAVED TRUST_CKTYPE
frames  total: 5799   initial: 0
streams: 2
Suggested BufferSize: 0
Size:  352 x 288
==========================
list_end=0xD4
==> Found video stream: 0
====== STREAM Header =====
Type: vids   FCC:  (0)
Flags: 0
Priority: 0   Language: 0
InitialFrames: 0
Rate: 250000/10000 = 25.000
Start: 0   Len: 5799
Suggested BufferSize: 29788
Quality 0
Sample size: 0
==========================
found 'bih', 40 bytes of 40
======= VIDEO Format ======
  biSize 40
  biWidth 352
  biHeight 288
  biPlanes 1
  biBitCount 24
  biCompression 1482049860='DIVX'
  biSizeImage 304128
===========================
Regenerating keyframe table for MPEG4 video
list_end=0x146
==> Found audio stream: 1
====== STREAM Header =====
Type: auds   FCC:  (0)
Flags: 0
Priority: 0   Language: 0
InitialFrames: 0
Rate: 44100/1152 = 38.281
Start: 0   Len: 8899
Suggested BufferSize: 835
Quality 0
Sample size: 0
==========================
found 'wf', 30 bytes of 18
======= WAVE Format =======
Format Tag: 85 (0x55)
Channels: 2
Samplerate: 44100
avg byte/sec: 14668
Block align: 1152
bits/sample: 0
cbSize: 12
mp3.wID=1
mp3.fdwFlags=0x2
mp3.nBlockSize=1152
mp3.nFramesPerBlock=1
mp3.nCodecDelay=0
===========================
list_end=0x125D754
Found movie at 0x80C - 0x125D754
Reading INDEX block, 14698 chunks for 5799 frames (fpos=0x125d75c)
AVI index offset: 0x808 (movi=0x80C idx0=0x4 idx1=0x74)
Auto-selected AVI audio ID = 1
Auto-selected AVI video ID = 0
AVI: Searching for audio stream (id:1)
AVI video size=15716220 (5799) audio size=3409782 (8899)
VIDEO:  [DIVX]  352x288  24bpp  25.000 fps  542.0 kbps (66.2 kbyte/s)
[V] filefmt:3  fourcc:0x58564944  size:352x288  fps:25.00  ftime:=0.0400
get_path('sub/') -> '/home/jan/.mplayer/sub/'
get_path('default.sub') -> '/home/jan/.mplayer/default.sub'
==========================================================================
Opening audio decoder: [mp3lib] MPEG layer-2, layer-3
dec_audio: Allocating 4608 + 65536 = 70144 bytes for output buffer.
mp3lib: made decode tables with MMX optimization
mp3lib: using MMX optimized decore!
MP3lib: init layer2&3 finished, tables done
MPEG 1.0, Layer III, 44100 Hz 32 kbit Joint-Stereo, BPF: 104
Channels: 2, copyright: No, original: Yes, CRC: No, emphasis: 0
AUDIO: 44100 Hz, 2 ch, 16 bit (0x10), ratio: 4000->176400 (32.0 kbit)
Selected audio codec: [mp3] afm:mp3lib (mp3lib MPEG layer-2, layer-3)
==========================================================================
X11 opening display: :0.0
vo: X11 color mask:  FFFFFF  (R:FF0000 G:FF00 B:FF)
vo: X11 running at 1152x864 with depth 24 and 32 bpp (":0.0" => local display)
[x11] Detected wm supports NetWM.
[x11] Detected wm supports FULLSCREEN state.
[x11] Detected wm supports ABOVE state.
[x11] Detected wm supports BELOW state.
[x11] Current fstype setting honours FULLSCREEN ABOVE BELOW X atoms
xscreensaver_disable: xscreensaver wid=10485761.
Detaching after fork from child process 12379.
Opening video filter: [scale w=512 h=384]
SwScale params: 512 x 384 (-1=no scaling)
==========================================================================
Opening video decoder: [ffmpeg] FFmpeg's libavcodec codec family
INFO: libavcodec init OK!
Selected video codec: [ffodivx] vfm:ffmpeg (FFmpeg MPEG-4)
==========================================================================
Checking audio filter chain for 44100Hz/2ch/16bit -> 44100Hz/2ch/16bit...
[libaf] Adding filter dummy
[dummy] Was reinitialized, rate=44100Hz, nch = 2, format = 0x00000001 and bps = 2
AF_pre: af format: 2 bps, 2 ch, 44100 hz, little endian signed int
AF_pre: 44100Hz 2ch Signed 16-bit (Little-Endian)
ao2: 44100 Hz  2 chans  Signed 16-bit (Little-Endian)
audio_setup: using '/dev/dsp' dsp device
audio_setup: using '/dev/mixer' mixer device
audio_setup: using 'pcm' mixer device
audio_setup: sample format: Signed 16-bit (Little-Endian) (requested: Signed 16-bit (Little-Endian))
audio_setup: using 2 channels (requested: 2)
audio_setup: using 44100 Hz samplerate (requested: 44100)
audio_setup: frags:   8/8  (8192 bytes/frag)  free:  65536
AO: [oss] 44100Hz 2ch Signed 16-bit (Little-Endian) (2 bps)
AO: Description: OSS/ioctl audio output
AO: Author: A'rpi
Building audio filter chain for 44100Hz/2ch/16bit -> 44100Hz/2ch/16bit...
[dummy] Was reinitialized, rate=44100Hz, nch = 2, format = 0x00000001 and bps = 2
[dummy] Was reinitialized, rate=44100Hz, nch = 2, format = 0x00000001 and bps = 2
Starting playback...
XXX initial  v_pts=0.000  a_pos=7978 (0.544)
[ffmpeg] aspect_ratio: 1.222222
VDec: vo config request - 352 x 288 (preferred csp: Planar YV12)
Trying filter chain: scale vo
VDec: using Planar YV12 as output csp (no 0)
Movie-Aspect is 1.22:1 - prescaling to correct movie aspect.
VO Config (352x288->352x288,flags=0,'MPlayer',0x32315659)
SwScaler: reducing / aligning filtersize 5 -> 4
SwScaler: reducing / aligning filtersize 5 -> 4
SwScaler: reducing / aligning filtersize 5 -> 4
SwScaler: reducing / aligning filtersize 5 -> 4

SwScaler: BICUBIC scaler, from Planar YV12 to Planar YV12 using MMX2
SwScaler: using 4-tap MMX scaler for horizontal luminance scaling
SwScaler: using 4-tap MMX scaler for horizontal chrominance scaling
SwScaler: using n-tap MMX scaler for vertical scaling (YV12 like)
SwScaler: 352x288 -> 512x384
REQ: flags=0x437  req=0x0
VO: [xmga] 512x384 => 512x418 Planar YV12
VO: Description: Matrox G200/G4x0/G550 overlay in X11 window (using /dev/mga_vid)
VO: Author: Zoltan Ponekker <pontscho at makacs.poliod.hu>
[mga] Using 3 buffers.
Using colorkey: ff00
[xmga] x: 320 y: 223 w: 512 h: 418
[xmga] dcx: 320 dcy: 223 dx: 0 dy: 0 dw: 512 dh: 418
*** [vo] Allocating mp_image_t, 512x384x12bpp YUV planar, 294912 bytes
*** [scale] Allocating (slices) mp_image_t, 352x288x12bpp YUV planar, 152064 bytes
[xmga] x: 5 y: 22 w: 512 h: 418
[xmga] dcx: 320 dcy: 223 dx: 0 dy: 0 dw: 512 dh: 418
*** [scale] Allocating (slices) mp_image_t, 352x288x12bpp YUV planar, 152064 bytes

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread -151139808 (LWP 12358)]
0x081b31d8 in swScale_MMX2 (c=Cannot access memory at address 0xfffffc08
) at swscale_template.c:2045
2045                    asm volatile(
(gdb) bt
#0  0x081b31d8 in swScale_MMX2 (c=Cannot access memory at address 0xfffffc08
) at swscale_template.c:2045
Cannot access memory at address 0xfffffc04
(gdb) disass $pc-32 $pc+32
Dump of assembler code from 0x81b31b8 to 0x81b31f8:
0x081b31b8 <swScale_MMX2+2804>: add    %cl,0x27bc(%ebp)
0x081b31be <swScale_MMX2+2810>: add    %al,(%eax)
0x081b31c0 <swScale_MMX2+2812>: movzww (%edx,%ebp,1),%ax
0x081b31c5 <swScale_MMX2+2817>: movzww 0x2(%edx,%ebp,1),%bx
0x081b31cb <swScale_MMX2+2823>: movq   (%ecx,%ebp,4),%mm1
0x081b31cf <swScale_MMX2+2827>: movq   0x8(%ecx,%ebp,4),%mm3
0x081b31d4 <swScale_MMX2+2832>: movd   (%esi,%eax,1),%mm0
0x081b31d8 <swScale_MMX2+2836>: movd   (%esi,%ebx,1),%mm2
0x081b31dc <swScale_MMX2+2840>: punpcklbw %mm7,%mm0
0x081b31df <swScale_MMX2+2843>: punpcklbw %mm7,%mm2
0x081b31e2 <swScale_MMX2+2846>: pmaddwd %mm1,%mm0
0x081b31e5 <swScale_MMX2+2849>: pmaddwd %mm2,%mm3
0x081b31e8 <swScale_MMX2+2852>: psrad  $0x8,%mm0
0x081b31ec <swScale_MMX2+2856>: psrad  $0x8,%mm3
0x081b31f0 <swScale_MMX2+2860>: packssdw %mm3,%mm0
0x081b31f3 <swScale_MMX2+2863>: pmaddwd %mm6,%mm0
0x081b31f6 <swScale_MMX2+2866>: packssdw %mm0,%mm0
End of assembler dump.
(gdb) info all-registers
eax            0xffff0000       -65536
ecx            0x884f4b8        142931128
edx            0x884a4a8        142910632
ebx            0x8840000        142868480
esp            0xfeffc76c       0xfeffc76c
ebp            0xfffffc00       0xfffffc00
esi            0xf6b9c040       -155598784
edi            0x884f8c8        142932168
eip            0x81b31d8        0x81b31d8
eflags         0x210203 2163203
cs             0x73     115
ss             0x7b     123
ds             0x7b     123
es             0x7b     123
fs             0x0      0
gs             0x33     51
st0            -nan(0x080808080)        (raw 0xffff0000000080808080)
st1            -nan(0x0fbc84438)        (raw 0xffff00000000fbc84438)
st2            -nan(0xfffffd68000a4979) (raw 0xfffffffffd68000a4979)
st3            -nan(0xfb4124e51fda)     (raw 0xffff0000fb4124e51fda)
st4            -nan(0x1717171717171717) (raw 0xffff1717171717171717)
st5            -nan(0x1717171717171717) (raw 0xffff1717171717171717)
st6            -nan(0x2000200020002)    (raw 0xffff0002000200020002)
st7            <invalid float value>    (raw 0xffff0000000000000000)
fctrl          0x37f    895
fstat          0x120    288
ftag           0xaaaa   43690
fiseg          0x73     115
fioff          0x80fb2cc        135246540
foseg          0x7b     123
fooff          0xfeffd25c       -16788900
fop            0x15d    349
xmm0           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x8000000000000000, 0x8000000000000000}, v16_int8 = {0xff <repeats 16 times>}, v8_int16 = {0xffff, 0xffff, 0xffff, 0xffff,
    0xffff, 0xffff, 0xffff, 0xffff}, v4_int32 = {0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff}, v2_int64 = {0xffffffffffffffff, 0xffffffffffffffff},
  uint128 = 0xffffffffffffffffffffffffffffffff}
xmm1           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x8000000000000000, 0x8000000000000000}, v16_int8 = {0xff <repeats 16 times>}, v8_int16 = {0xffff, 0xffff, 0xffff, 0xffff,
    0xffff, 0xffff, 0xffff, 0xffff}, v4_int32 = {0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff}, v2_int64 = {0xffffffffffffffff, 0xffffffffffffffff},
  uint128 = 0xffffffffffffffffffffffffffffffff}
xmm2           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x8000000000000000, 0x8000000000000000}, v16_int8 = {0xff <repeats 16 times>}, v8_int16 = {0xffff, 0xffff, 0xffff, 0xffff,
    0xffff, 0xffff, 0xffff, 0xffff}, v4_int32 = {0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff}, v2_int64 = {0xffffffffffffffff, 0xffffffffffffffff},
  uint128 = 0xffffffffffffffffffffffffffffffff}
xmm3           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x8000000000000000, 0x8000000000000000}, v16_int8 = {0xff <repeats 16 times>}, v8_int16 = {0xffff, 0xffff, 0xffff, 0xffff,
    0xffff, 0xffff, 0xffff, 0xffff}, v4_int32 = {0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff}, v2_int64 = {0xffffffffffffffff, 0xffffffffffffffff},
  uint128 = 0xffffffffffffffffffffffffffffffff}
xmm4           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x8000000000000000, 0x8000000000000000}, v16_int8 = {0xff <repeats 16 times>}, v8_int16 = {0xffff, 0xffff, 0xffff, 0xffff,
    0xffff, 0xffff, 0xffff, 0xffff}, v4_int32 = {0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff}, v2_int64 = {0xffffffffffffffff, 0xffffffffffffffff},
  uint128 = 0xffffffffffffffffffffffffffffffff}
xmm5           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x8000000000000000, 0x8000000000000000}, v16_int8 = {0xff <repeats 16 times>}, v8_int16 = {0xffff, 0xffff, 0xffff, 0xffff,
    0xffff, 0xffff, 0xffff, 0xffff}, v4_int32 = {0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff}, v2_int64 = {0xffffffffffffffff, 0xffffffffffffffff},
  uint128 = 0xffffffffffffffffffffffffffffffff}
xmm6           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x8000000000000000, 0x8000000000000000}, v16_int8 = {0xff <repeats 16 times>}, v8_int16 = {0xffff, 0xffff, 0xffff, 0xffff,
    0xffff, 0xffff, 0xffff, 0xffff}, v4_int32 = {0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff}, v2_int64 = {0xffffffffffffffff, 0xffffffffffffffff},
  uint128 = 0xffffffffffffffffffffffffffffffff}
xmm7           {v4_float = {0x0, 0x0, 0x0, 0x0}, v2_double = {0x8000000000000000, 0x8000000000000000}, v16_int8 = {0xff <repeats 16 times>}, v8_int16 = {0xffff, 0xffff, 0xffff, 0xffff,
    0xffff, 0xffff, 0xffff, 0xffff}, v4_int32 = {0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff}, v2_int64 = {0xffffffffffffffff, 0xffffffffffffffff},
  uint128 = 0xffffffffffffffffffffffffffffffff}
mxcsr          0x1f80   8064
mm0            {uint64 = 0x80808080, v2_int32 = {0x80808080, 0x0}, v4_int16 = {0x8080, 0x8080, 0x0, 0x0}, v8_int8 = {0x80, 0x80, 0x80, 0x80, 0x0, 0x0, 0x0, 0x0}}
mm1            {uint64 = 0xfbc84438, v2_int32 = {0xfbc84438, 0x0}, v4_int16 = {0x4438, 0xfbc8, 0x0, 0x0}, v8_int8 = {0x38, 0x44, 0xc8, 0xfb, 0x0, 0x0, 0x0, 0x0}}
mm2            {uint64 = 0xfffffd68000a4979, v2_int32 = {0xa4979, 0xfffffd68}, v4_int16 = {0x4979, 0xa, 0xfd68, 0xffff}, v8_int8 = {0x79, 0x49, 0xa, 0x0, 0x68, 0xfd, 0xff, 0xff}}
mm3            {uint64 = 0xfb4124e51fda, v2_int32 = {0x24e51fda, 0xfb41}, v4_int16 = {0x1fda, 0x24e5, 0xfb41, 0x0}, v8_int8 = {0xda, 0x1f, 0xe5, 0x24, 0x41, 0xfb, 0x0, 0x0}}
mm4            {uint64 = 0x1717171717171717, v2_int32 = {0x17171717, 0x17171717}, v4_int16 = {0x1717, 0x1717, 0x1717, 0x1717}, v8_int8 = {0x17, 0x17, 0x17, 0x17, 0x17, 0x17, 0x17, 0x17}}
mm5            {uint64 = 0x1717171717171717, v2_int32 = {0x17171717, 0x17171717}, v4_int16 = {0x1717, 0x1717, 0x1717, 0x1717}, v8_int8 = {0x17, 0x17, 0x17, 0x17, 0x17, 0x17, 0x17, 0x17}}
mm6            {uint64 = 0x2000200020002, v2_int32 = {0x20002, 0x20002}, v4_int16 = {0x2, 0x2, 0x2, 0x2}, v8_int8 = {0x2, 0x0, 0x2, 0x0, 0x2, 0x0, 0x2, 0x0}}
mm7            {uint64 = 0x0, v2_int32 = {0x0, 0x0}, v4_int16 = {0x0, 0x0, 0x0, 0x0}, v8_int8 = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}}
(gdb)


Fedora Core 2
Linux 2.6.8
/lib/libc-2.3.3.so
Reading specs from /extra/gcc295/lib/gcc-lib/i686-pc-linux-gnu/2.95.3/specs
gcc version 2.95.3 20010315 (release)
GNU ld version 2.15.90.0.3 20040415
GNU assembler 2.15.90.0.3 20040415

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 8
model name      : Pentium III (Coppermine)
stepping        : 3
cpu MHz         : 770.007
cache size      : 256 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 mmx fxsr sse
bogomips        : 1523.71