[FFmpeg-devel] [PATCH] get rid of nonsense palette pointer for RGB8, GRAY8 etc.

[Reimar Döffinger]

On Fri, Mar 20, 2009 at 11:33:11PM +0100, Michael Niedermayer wrote:
> On Fri, Mar 20, 2009 at 09:43:23PM +0100, Reimar D?ffinger wrote:
> > On Fri, Mar 20, 2009 at 09:12:49PM +0100, Michael Niedermayer wrote:
> > > about the avpicture_layout patch i dont understand it, avpicture_get_size()
> > > should return the correct size
> > 
> > avpicture_get_size() returns the size needed for the memory buffer, that
> > includes a palette even for GRAY8.
> 
> hmm avpicture_get_size)( use for malloc needs to include it the
> muxer stuff should not, so i guess changing avpicture_get_size() is ok

Huh? I don't completely understand what you mean.
You mean avpicture_get_size should return the size that the frame should
allocate as raw data?
For PAL8, should that include the alignment for the palette or not?
It's also a bit ugly because avpicture_get_size is part of the public
API, though the description is vague enough that both behaviours match
it...
Attached patch would do that, though it does not improve how PAL8 is
treated (palette is 4-byte aligned, palette is written as native-endian).
-------------- next part --------------
Index: libavcodec/imgconvert.c
===================================================================
--- libavcodec/imgconvert.c	(revision 18076)
+++ libavcodec/imgconvert.c	(working copy)
@@ -786,6 +786,17 @@
 int avpicture_get_size(int pix_fmt, int width, int height)
 {
     AVPicture dummy_pict;
+    if(avcodec_check_dimensions(NULL, width, height))
+        return -1;
+    switch (pix_fmt) {
+    case PIX_FMT_RGB8:
+    case PIX_FMT_BGR8:
+    case PIX_FMT_RGB4_BYTE:
+    case PIX_FMT_BGR4_BYTE:
+    case PIX_FMT_GRAY8:
+        // do not include palette for these pseudo-paletted formats
+        return width * height;
+    }
     return avpicture_fill(&dummy_pict, NULL, pix_fmt, width, height);
 }
 
@@ -1125,7 +1136,7 @@
     int size;
     void *ptr;
 
-    size = avpicture_get_size(pix_fmt, width, height);
+    size = avpicture_fill(picture, NULL, pix_fmt, width, height);
     if(size<0)
         goto fail;
     ptr = av_malloc(size);
Index: tests/libav.regression.ref
===================================================================
--- tests/libav.regression.ref	(revision 18076)
+++ tests/libav.regression.ref	(working copy)
@@ -47,7 +47,7 @@
 ./tests/data/b-pbmpipe.pbm CRC=0x806e17d8
 aff140ce80a1c86c1bf54118ad23da7b *./tests/data/b-pgmpipe.pgm
 2534775 ./tests/data/b-pgmpipe.pgm
-./tests/data/b-pgmpipe.pgm CRC=0x0e82c482
+./tests/data/b-pgmpipe.pgm CRC=0xf485870f
 9169b1f1ca56f01a6e1f5041572aa1d4 *./tests/data/b-ppmpipe.ppm
 7603575 ./tests/data/b-ppmpipe.ppm
 ./tests/data/b-ppmpipe.ppm CRC=0x79bd6ce6
@@ -56,7 +56,7 @@
 b977a4fedff90a79baf70c8e02986820 *./tests/data/b-libav.y4m
 3801810 ./tests/data/b-libav.y4m
 0a6d74b54396884f117669965b57d3b5 *./tests/data/b-libav02.pgm
-./tests/data/b-libav%02d.pgm CRC=0xc8032eb1
+./tests/data/b-libav%02d.pgm CRC=0x7e552eb1
 101391 ./tests/data/b-libav02.pgm
 dbe42bd8d9ca0acbd2673bd739705f0f *./tests/data/b-libav02.ppm
 ./tests/data/b-libav%02d.ppm CRC=0x6f775c0d