[FFmpeg-devel] Fix MMX dct_quantize for non zigzag_direct scans

[Ramiro Polla]

Hello,

If I use dct_quantize_c for the mimic encoder, all goes well. But if I
use MMX dct_quantize, it doesn't work the same. The last_non_zero value
returned is wrong. There is a note about it in
libavcodec/mpegvideo_enc.c and it searches again for the last_non_zero
coeff after all calls to dct_quantize if they're not dct_quantize_c.

MMX dct_quantize uses inv_zigzag_direct16 independing of the scantable,
so that's why (I think) it returns the wrong value. If I change
ScanTable to also have an inverse[64] table on MMX and use that instead,
it returns the right value. But MMX dct_quantize uses custom code to
permute the block in the end, and with this new last_non_zero value
(from inverse[]) it doesn't copy the blocks correctly. If I add
non-optimized block permuting code for the cases of alternate scan, it
works again.

I thought maybe the current code always returned a larger last_non_zero
value, but it doesn't happen all the time (although it happens most of
the time).

But then I got confused... Isn't that custom permute code only good for
ff_zigzag_direct? Shouldn't it copy the wrong values for alternate
scans? Or are the values being permuted somewhere else again?

Anyways, attached patch gives same resulting file for tempete.cif
encoded in mpeg4 with -flags +alt.

Ramiro Polla

-------------- next part --------------
A non-text attachment was scrubbed...
Name: i3862.diff
Type: text/x-patch
Size: 4564 bytes
Desc: not available
URL: <http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/attachments/20080513/79180655/attachment.bin>