[FFmpeg-devel] avpicture_layout() writes passed the buffer length for pseudo-paletted formats (bug)

[Matthew Einhorn]

On Thu, Aug 25, 2011 at 10:21 AM, Stefano Sabatini
<stefano.sabatini-lala at poste.it> wrote:
> On date Sunday 2011-08-21 14:09:12 -0400, Matthew Einhorn encoded:
>> Hi,
>>
>> The problem is that in the docs avpicture_layout() says it writes a
>> picture with size avpicture_get_size(). So one would make the buffer
>> input to avpicture_layout() of that size. However, for pseudo-paletted
>> formats (as listed in avpicture_get_size()) the amount written to
>> buffer is avpicture_get_size()+1024 (which is also the
>> avpicture_fill() size).
>>
>> If the buffer size is less than avpicture_get_size() (as indicated by
>> dest_size in avpicture_layout()) an error is returned by
>> avpicture_layout(). However, for these pseudo-paletted formats no
>> error is returned if the buffer is less than avpicture_get_size()+1024
>> and the function simply writes passed the actual buffer length for a
>> total of avpicture_get_size()+1024.
>>
>> The reason for this it seems is that avpicture_layout() wasn't
>> supposed to write the palette for these pseudo-paletted formats, but
>> the function still writes it resulting in the problem.
>>
>>
>> The solution is to:
>> *Either remove the exception for pseudo-paletted formats so that
>> avpicture_get_size() now completely relies on avpicture_fill() (will
>> probably break backwards compatibility) and both return the same size.
>> *Make an exception in avpicture_layout() as well for the
>> pseudo-paletted formats so that the palette is not written for them.
>> *If this "is" intended behavior update avpicture_layout() docs to
>> indicate that for these formats the buffer needs to be larger than
>> avpicture_get_size() by 1024.
>
>> *Or maybe there's some other deeper issue here (other then to prevent
>> copying of a standard palette) as to why these pseudo-paletted formats
>> are treated specially?
>
> Best solution is to get rid of these pseudo-palette hacks altogheter,
> please wait since I have some cached patches for that.

Ok,
Thanks
Matt