[Ffmpeg-devel] swscale and palette ... what am I missing?

[Michael Niedermayer]

Hi

On Tue, Feb 27, 2007 at 11:33:02PM +0100, Karl H. Beckers wrote:
> Hi all,
> 
> I'm trying to convert a pal8 frame to yuv420p using libswscale rather
> than converting it to rbg myself. From what I'm reading in the archives
> I kinda think this should be possible, but I'm not getting past a
> segfault in swScale_MMX with input_pixfmt = PIX_FMT_PAL8 where RGBx work
> nicely.
> 
> I'm setting up my frame like the following, contiguous char data in
> data[0], linesize in bytes in linesize[0] and pointer to palette as arbg
> (with alpha always 0) in data[1]:
>             avpicture_fill ((AVPicture *) p_inpic, scratchbuf8bit,
>                             input_pixfmt, image->width, image->height);
>             p_inpic->data[0] = scratchbuf8bit;
>             p_inpic->linesize[0] = image->width;
>             p_inpic->data[1] = job->color_table;
> 
> then I construct my SwsContext:
>             img_resample_ctx = sws_getContext (image->width,
>                                                image->height,
>                                                input_pixfmt,
>                                                out_st->codec->width,
>                                                out_st->codec->height,
>                                                out_st->codec->pix_fmt,1,
>                                                NULL, NULL, NULL);
> 
> and later call the sws_scale function:
>     if (sws_scale (img_resample_ctx, p_inpic->data, p_inpic->linesize,
>                    0, image->height, p_outpic->data, p_outpic->linesize)
>  		< 0) {
>         fprintf (stderr, .......
> 
> 
> this always yields a segfault immediately:
> 
> Program terminated with signal 11, Segmentation fault.
> #0  0x0836dd0f in swScale_MMX (c=0x87f2ca0, src=0x87c5490,
> srcStride=0xb639228c, srcSliceY=0, srcSliceH=144, dst=0x87c5570,
> dstStride=0xb6392298)
>     at swscale_template.c:2288
> 2288                    int b= pal[d]     &0xFF;
> 
> 
> Trying to debug this:
> 
> Starting program: /home/kb87850/xvidcap/bin/xvidcap 
> [Thread debugging using libthread_db enabled]
> [New Thread -1221232448 (LWP 7570)]
> [New Thread -1238369376 (LWP 7571)]
> [mpeg4 @ 0x8444f60]removing common factors from framerate
> [New Thread -1248855136 (LWP 7572)]
> [Switching to Thread -1238369376 (LWP 7571)]
> 
> Breakpoint 1, swScale_MMX (c=0xb5950fc0, src=0xb5935910,
> srcStride=0xb62ff28c, srcSliceY=0, srcSliceH=144, dst=0xb593c9e0,
> dstStride=0xb62ff298)
>     at swscale_template.c:2285
> 2285            for(i=0; i<width; i++)
> (gdb) print i
> $7 = <value optimized out>
> (gdb) print width
> No symbol "width" in current context.
> (gdb) step
> 2288                    int b= pal[d]     &0xFF;
> (gdb) print d
> No symbol "d" in current context.
> (gdb) print src[0]
> $8 = (uint8_t *) 0xb59359d8 ""
> (gdb) print /d src[0]
> $9 = 3046332888
> (gdb) print pal
> $10 = (uint8_t *) 0x0
> (gdb) print *pal
> Cannot access memory at address 0x0
> (gdb) print pal[3046332888]
> $11 = 0 '\0'
> 
> 
> comparing with the relevant code in swscale_template.c:
> 
> static inline void RENAME(palToY)(uint8_t *dst, uint8_t *src, int width,
> uint32_t *pal)
> {
>         int i;
>         for(i=0; i<width; i++) 
>         {    
>                 int d= src[i];
>                 int b= pal[d]     &0xFF;
>                 int g=(pal[d]>>8 )&0xFF;
>                 int r= pal[d]>>16;
> av_log(NULL, AV_LOG_DEBUG, "i: %i\n", i);
> 
>                 dst[i]= ((RY*r + GY*g + BY*b)>>RGB2YUV_SHIFT) + 16;
>         }    
> }
> 
> in gdb I see neither width nor stepping through the program see the line
> "int d= src[i];" at all. pal seems to be a NULL pointer, but I don't
> know why ... the palette needs to go in AVFrame.data[1] after all,
> doesn't it?
> 
> Am I missing anything obvious?

hmm dunno, very quickly looking at your mail it looks all ok, i suggest you
place a few random av_log() in the code to see where the non zero pal becomes
NULL

and of course a patch to fix this is very welcome if its a bug :)

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Why not whip the teacher when the pupil misbehaves? -- Diogenes of Sinope
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20070228/5293bcc2/attachment.pgp>