[MPlayer-users] [BUGREPORT] VP6 / NSV

[Martin Simmons]

>>>>> On Sun, 12 Sep 2004 18:36:43 +0200, Reimar Döffinger <Reimar.Doeffinger at stud.uni-karlsruhe.de> said:

  >>> First, configure MPlayer as usual without debugging.  Then open
  >>> config.mak, search for "OPTFLAGS =" and replace the "-O4" in
  >>> those flags by "-O4 -g", recompile and try if the problem persists. If
  >>> yes, send another bugreport as you now hav proper debugging info.
  >> 
  >> Done, and the problem persists...
  >> 
  >> GDB output attached.
  >> 
  >> 
  >>> Now do the same again, but instead of replacing "-O4" with "-O4 -g" 
  >>> replace it by "-O2".
  >> 
  >> That's odd.  Changing O4 to O2 seems to solve the problem entirely.

  Reimar> That's what I expected.
  Reimar> I'd like you to do "gdb mplayer" and on the gdb commandline
  Reimar> disassemble extend_stack_for_dll_alloca
  Reimar> and post the output here.
  Reimar> Please try the attached patch, in case the compiler optimizes a bit too 
  Reimar> good, declaring volatile should avoid it.

The problem is that extend_stack_for_dll_alloca() gets inlined into
PE_InitDLL(), so the extended stack space is still *above* esp when the DLL
entrypoint is called:

	movl	16(%ebp), %eax
	subl	$131072, %esp       // alloca(0x20000)
	movl	$4660, 16(%esp)     // *(int*)mem=0x1234
	movl	%ecx, (%esp)
	movl	%eax, 8(%esp)
	movl	12(%ebp), %eax
	movl	%eax, 4(%esp)
	call	*%edx               // call entry


The following patch fixes it for gcc 3.2.2 at least:

--- pe_image.c~	2002-09-13 20:43:13.000000000 +0100
+++ pe_image.c	2004-09-12 22:53:39.000000000 +0100
@@ -896,7 +896,7 @@
  * The extend_stack_for_dll_alloca() routine just preallocates a big chunk
  * of memory on the stack, for use by the DLLs alloca routine.
  */
-static void extend_stack_for_dll_alloca(void)
+__attribute_noinline__ static void extend_stack_for_dll_alloca(void)
 {
 #ifndef __FreeBSD__
     void* mem=alloca(0x20000);

__Martin