[MPlayer-users] OpenGL freezes X with ATI fglrx driver on x86_64

[Karthik Arumugham]

Hello,

I'm experiencing problems with OpenGL output. I'm running mplayer as  
mplayer -vo gl2:yuv=3 (or gl, both freeze in the same way). The first  
time I run mplayer, it runs fine. The second time I run mplayer, my X  
server locks up, and the only way to recover is to ssh in from another  
machine, kill the X server, and rmmod the ATI fglrx video driver  
kernel module so it can be reloaded when X restarts.

I'm using an ATI X1300XT on an x86_64 Ubuntu Gutsy system, kernel  
2.6.23.9, ATI proprietary driver 7.11 (I've tried with older versions  
of the driver and various 2.6.22 kernels with the same results.) I've  
tried mplayer 1.0rc2 and the current svn version with the same  
results. Playing with xorg.conf ATI driver options has no effect.  
Unfortunately, I can't use xv output because of video tearing issues  
due to ATI's subpar drivers, and X11 output both has slightly glitchy  
video and is too slow to keep up with 1920x1080 MPEG streams, even on  
a 2.4GHz Core 2 Duo. (ATI's wonderful drivers at work again.)

A backtrace of mplayer after X freezes shows mplayer stuck in a poll()  
with the following:
(gdb) bt
#0  0x00002b59bdcaadef in poll () from /lib/libc.so.6
#1  0x00002b59bb55b470 in ?? () from /usr/lib/libX11.so.6
#2  0x00002b59bb55b899 in _XRead () from /usr/lib/libX11.so.6
#3  0x00002b59bb55c2c1 in _XReply () from /usr/lib/libX11.so.6
#4  0x00002b59bbc6bcf9 in ?? () from /usr/lib/libGL.so.1.2
#5  0x00002b59bbc6ae27 in glXMakeCurrentReadSGI () from /usr/lib/ 
libGL.so.1.2
#6  0x0000000000477a64 in setGlWindow (vinfo=0xcdb110,  
context=0xcdb118, win=46137346)
    at gl_common.c:1534
#7  0x0000000000476f2f in config (width=<value optimized out>,  
height=<value optimized out>,
    d_width=1280, d_height=720, flags=0, title=0x9328e5 "MPlayer",  
format=842094169) at vo_gl2.c:632
#8  0x0000000000469974 in config_video_out (vo=0xcb0280, width=1280,  
height=720, d_width=4294967295,
    d_height=4294967295, flags=0, title=0x9328e5 "MPlayer",  
format=842094169) at video_out.c:352
#9  0x00000000004cf463 in config (vf=0xe7f600, width=1280, height=720,  
d_width=1280, d_height=720,
    flags=0, outfmt=842094169) at vf_vo.c:65
#10 0x00000000004a7c86 in vf_config_wrapper (vf=0xe7f600, width=1,  
height=-1, d_width=-1,
    d_height=46137346, flags=46137346, outfmt=4294967292) at vf.c:617
#11 0x00000000004a63e0 in mpcodecs_config_vo (sh=0xe46b10, w=<value  
optimized out>,
    h=<value optimized out>, preferred_outfmt=<value optimized out>)  
at vd.c:305
#12 0x00000000004dda08 in decode (sh=0xe46b10, data=0x2b59bb19b010,  
len=<value optimized out>,
    flags=<value optimized out>) at vd_libmpeg2.c:179
#13 0x00000000004a541b in decode_video (sh_video=0xe46b10,  
start=0x2b59bb19b010 "", in_size=64925,
    drop_frame=0, pts=0.016683332622051239) at dec_video.c:366
#14 0x0000000000442fe8 in main (argc=<value optimized out>,  
argv=<value optimized out>) at mplayer.c:2049

Now, to make things more interesting, I can actually run the Ubuntu- 
supplied mplayer 1.0rc1 with -vo gl2:yuv=5 repeatedly without  
crashing. Unfortunately, yuv=5 does not provide for gamma control, so  
my blacks turn into grays rendering videos unwatchable. (I've tried  
the vf eq2 settings, but tweaked as best I can they are still  
significantly inferior to using yuv=3.) Building my own 1.0rc1 results  
in the same freezing after the first run as with newer versions. Yet  
the Ubuntu diffs for mplayer do not touch vo_gl.c or vo_gl2.c, so it  
must have something to do with another patch or how it's being  
configured/built (I'm just using "--enable-debug=3" for my builds; it  
crashes just the same without debugging enabled.) But the only thing  
the Ubuntu build configure does is either enable/disable the gui and  
enable/disable building mencoder.

And finally, I can run mplayer 1.0rc1 (either Ubuntu-supplied or built  
on my own) with yuv=3, and it crashes before any video is output  
(without breaking X):
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 47824830669152 (LWP 26324)]
0x0000000000cb7940 in ?? ()
(gdb) bt
#0  0x0000000000cb7940 in ?? ()
#1  0x0000000000443839 in initGl (d_width=1280, d_height=720) at  
vo_gl2.c:770
#2  0x00000000004441b9 in config (width=<value optimized out>,  
height=<value optimized out>,
    d_width=1280, d_height=720, flags=0, title=0x833311 "MPlayer",  
format=842094169) at vo_gl2.c:882
#3  0x0000000000488d82 in config (vf=0xcb1530, width=1280, height=720,  
d_width=1280, d_height=720,
    flags=0, outfmt=842094169) at vf_vo.c:69
#4  0x0000000000460125 in vf_config_wrapper (vf=0xcb1530,  
width=12200192, height=13343856,
    d_width=-29331, d_height=14494080, flags=2155905152,  
outfmt=13334848) at vf.c:627
#5  0x000000000045c346 in mpcodecs_config_vo (sh=0xc77e80, w=<value  
optimized out>,
    h=<value optimized out>, preferred_outfmt=<value optimized out>)  
at vd.c:309
#6  0x000000000045ded6 in decode (sh=0xc77e80, data=0x2b7f1114c010,  
len=<value optimized out>,
    flags=<value optimized out>) at vd_libmpeg2.c:179
#7  0x000000000045b336 in decode_video (sh_video=0xc77e80,  
start=0x2b7f1114c010 "", in_size=64925,
    drop_frame=0, pts=0.016683332622051239) at dec_video.c:357
#8  0x0000000000412af9 in main (argc=4, argv=0x0) at mplayer.c:4171

I'm also running mplayer on a similar x86_64 Ubuntu Gutsy machine—but  
with an nVidia card, with no problems. So this does seem to be ATI- 
driver specific. Anyone have any clue what might be going on here, or  
have any success getting OpenGL output working with ATI's fglrx driver?