[MPlayer-dev-eng] raw filter (ve_raw.c) writing improper frame data

[jkoleszar at on2.com]

Hi all,

I'm having some problems when using the raw video encoder from mencoder.
Basically, I'm trying to transcode a RLE encoded Quicktime file to raw
BGRA frames, but the resulting frames are stored with the wrong pitch. My
source material is 756x203. This gets expanded internally to a 768x208
buffer, to align the width and heights to multiples of 16. When the raw
filter calls the muxer (put_image() in ve_raw.c) it writes the expanded
image rather than a packed version of the display image, writing frames
that are 638976 bytes rather than the expected 613872.

Is the right way to fix this changing ve_raw.c to properly support images
with variable strides (mp_image_t.width > mp_image_t.w) or is it a bug
that these types of images are even being passed to this filter, since it
doesn't advertise support (ie, it doesn't set VFCAP_ACCEPT_STRIDE).

Both of these command lines exhibit this problem:

1) mencoder -ovc raw -o test_no_format.avi betina.mov
2) mencoder -ovc raw -vf format=bgra -o test_format_bgra.avi betina.mov

This command line does not exhibit the problem:

3) mencoder -ovc raw -vf format=yv12 -o test_format_yv12.avi betina.mov

It's worth noting that BGRA is the output colorspace of the QT RLE decoder.

I'm trying to figure out what the right place to fix this bug is. If the
right solution is to handle these images within the raw filter, I can
handle that without much trouble. It seems to me that the raw filter
shouldn't be passed these types of images and that the fix is probably at
a higher level, but if someone can point me in the right direction, it'd
be appreciated.

Thanks a lot,

John Koleszar