[MPlayer-dev-eng] amd64 -sws 0 fix (iow: playing with NX bit)

[Aurelien Jacobs]

Hi,

Reimar pointed out a bug in swscaler introduced by my amd64 patch.
This leaded me to do some sws tests on amd64. And I found out that
upscaling with -sws 0 crashed. There was 2 reason for this. First
there was bugs in my registers handling. The attached patch fix
this, and I will commit it soon.

The second reason is more problematic. This scaler use some code
which is generated at runtime (the funnyCode). The generated code
is simply written in a char[].
Now, remember we are speeking of amd64. And remember that shiny
new feature introduced by amd64, which is called NX bit....
Yes, you've got it. The CPU simply prevent the execution of
this code which is in a data segment, causing a segfault.

The only way I found to be able to use this, is to boot the kernel
with noexec=off parameter, thus disabling the use of this NX bit.
Theoricaly another solution would be to use `execstack -s mplayer`
to teach the kernel that it should not use the NX bit for this
program, but it didn't worked for me.

Now the question is how to solve this, cleanly ?
I can see few options :
(1) Find how to make execstack working (or any other way to set
    executable stack flag, it seems that the linker is also able
    to do it).
(2) Rewrite this code so that it don't need to be generated at
    runtime. This would be clean but this would be slower and
    thus is not acceptable.
(3) Simply document that -sws 0 need to disable NX bit support on
    CPUs which have it.
(4) Runtime detection of the NX bit, and clean exit with a
    clear message if -sws 0 is used.
(5) Probably other solutions, maybe better...

What do you think about this ?

Aurel
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: sws0_x86_64.diff
URL: <http://lists.mplayerhq.hu/pipermail/mplayer-dev-eng/attachments/20050120/e9be46fd/attachment.asc>