[Ffmpeg-devel] swscale and 64 bit problem (mmx code)

[Reimar Döffinger]

Hello,
On Thu, Mar 01, 2007 at 01:18:12PM +0000, Giancarlo Formicuccia wrote:
> libswscale seems to be unsafe on 64 bit platforms. The problem is 
> the 32-bit chrMmxFilter array of SwsContext, because it stores
> pointers to allocated memory (lumMmxFilter has probably the same
> issue).
[...]
> Fixing the code is not trivial, due to the many hardcoded values in the asm code...
> Ideas?

Well, the non-SWS_ACCURATE_RND case is trivial to fix I think, see attached
patch.

Greetings,
Reimar D?ffinger
-------------- next part --------------
Index: libswscale/swscale_template.c
===================================================================
--- libswscale/swscale_template.c	(revision 22381)
+++ libswscale/swscale_template.c	(working copy)
@@ -3168,14 +3168,16 @@
             }else{
 		for(i=0; i<vLumFilterSize; i++)
 		{
-			lumMmxFilter[4*i+0]= (int32_t)lumSrcPtr[i];
+			lumMmxFilter[4*i+0]= (uintptr_t)lumSrcPtr[i];
+			lumMmxFilter[4*i+1]= (uintptr_t)lumSrcPtr[i] >> 32;
 			lumMmxFilter[4*i+2]= 
 			lumMmxFilter[4*i+3]= 
 				((uint16_t)vLumFilter[dstY*vLumFilterSize + i])*0x10001;
 		}
 		for(i=0; i<vChrFilterSize; i++)
 		{
-			chrMmxFilter[4*i+0]= (int32_t)chrSrcPtr[i];
+			chrMmxFilter[4*i+0]= (uintptr_t)chrSrcPtr[i];
+			chrMmxFilter[4*i+1]= (uintptr_t)chrSrcPtr[i] >> 32;
 			chrMmxFilter[4*i+2]= 
 			chrMmxFilter[4*i+3]= 
 				((uint16_t)vChrFilter[chrDstY*vChrFilterSize + i])*0x10001;