[MPlayer-dev-eng] xvmc and nvidia twinview

[Ilia Mirkin]

There was a thread earlier on about xvmc getting a BadMatch error, and
it later transpired that the person was using twinview, and it fizzled
out without a solution. I find myself in the same situation, however I
noticed that xine has no problem working with xvmc.

The problem stems from mplayer assuming that there is an XV_COLORKEY
setting on the adapter. Now, though I admit to not having a clue as to
what that parameter means, it seems that if I remove the lines that set
the colorkey information, mplayer works with xvmc just fine.

Here is some information about my setup,

X-Video Extension version 2.2
screen #0
  Adaptor #0: "NV17 Video Texture"
    number of ports: 1
    port base: 145
    operations supported: PutImage 
    supported visuals:
...
    number of attributes: 3
      "XV_SET_DEFAULTS" (range 0 to 0)
              client settable attribute
      "XV_ITURBT_709" (range 0 to 1)
              client settable attribute
              client gettable attribute (current value is 0)
      "XV_SYNC_TO_VBLANK" (range 0 to 1)
              client settable attribute
              client gettable attribute (current value is 1)
    maximum XvImage size: 2046 x 2046

and here is the change that I made that makes xvmc seemingly work fine:

in libvo/vo_xvmc.c, the end of init_keycolor becomes

   keycolor_handling = AUTOPAINT_COLORKEY;

instead of

   rez=XvGetPortAttribute(mDisplay,xv_port, xv_atom, &colorkey);
   if(rez == Success){
      keycolor = colorkey;
      if(keycolor_handling == AUTO_COLORKEY){
         keycolor_handling = MANUALFILL_COLORKEY;
      }
   }

This is of course an improper fix in the long term, but I hope that
someone who understands more about this than I can use this information
to make a proper patch. (e.g. one might query the existence of
XV_COLORKEY first, of maybe the last value of the XInternAtom should be
True... the real problem is that I have no idea what should be done with
the keycolor setting.)

A note about my hardware: GeForceFX 5200, TwinView, 1.0-6629 drivers.