[FFmpeg-devel] [PATCH] libswscale: Fix the anonymous memory mapping in sws_GetContext for NetBSD when MMX2 optimisations are enabled.

[Grant Carver]

Hi there

Creation of a swscale context was failing on NetBSD due to the anonymous
mmap call failing. It seems that if the MAP_ANONYMOUS flag is specified some
BSD flavours require the file descriptor to be -1, whilst on linux this
parameter is just ignored. 

This was picked up when creating a context for scaling planar YUV420 @
176x144 to RGB32 @ 352x288 with MMX2 optimisations enabled.

Attached is a patch which just changes the file descriptors to -1. This
appears to be ok as the parameter is a signed int, and is ignored on the
majority anyway. 

Regards
Grant


Index: libswscale/utils.c
===================================================================
--- libswscale/utils.c	(revision 31971)
+++ libswscale/utils.c	(working copy)
@@ -897,8 +897,8 @@
             c->chrMmx2FilterCodeSize = initMMX2HScaler(c->chrDstW,
c->chrXInc, NULL, NULL, NULL, 4);
 
 #ifdef MAP_ANONYMOUS
-            c->lumMmx2FilterCode = mmap(NULL, c->lumMmx2FilterCodeSize,
PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);
-            c->chrMmx2FilterCode = mmap(NULL, c->chrMmx2FilterCodeSize,
PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, 0, 0);
+            c->lumMmx2FilterCode = mmap(NULL, c->lumMmx2FilterCodeSize,
PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
+            c->chrMmx2FilterCode = mmap(NULL, c->chrMmx2FilterCodeSize,
PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
 #elif HAVE_VIRTUALALLOC
             c->lumMmx2FilterCode = VirtualAlloc(NULL,
c->lumMmx2FilterCodeSize, MEM_COMMIT, PAGE_EXECUTE_READWRITE);
             c->chrMmx2FilterCode = VirtualAlloc(NULL,
c->chrMmx2FilterCodeSize, MEM_COMMIT, PAGE_EXECUTE_READWRITE);
-------------- next part --------------
A non-text attachment was scrubbed...
Name: swscale_mmap_patch.diff
Type: application/octet-stream
Size: 1113 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20100819/b64f6813/attachment.obj>