[FFmpeg-devel] [PATCH][RFC] Lagarith Decoder.

[Nathan Caldwell]

2009/9/5 Loren Merritt <lorenm at u.washington.edu>:
> On Tue, 1 Sep 2009, Nathan Caldwell wrote:
>
>> Here are the latest lagarith patches.
>
> Your decoder disagrees with the official version on the 2nd frame of http://akuvian.org/stuff/lagarith_bork.avi

Finally got a chance to look into this, and found the problem, but I'm
not sure how to fix it.
What happens is on this clip when the probabilities get scaled in
lag_read_prob_header(), one of the values gets rounded up instead of
rounding down like reference does. For the offending value reference
gives 0x6ff while the code below gives 0x700.

        for (i = 1; i < 257; i++) {
                rac->prob[i] = ((uint64_t) rac->prob[i] <<
scale_factor) / cumul_prob;
            scaled_cumul_prob += rac->prob[i];
        }

Subtracting 1 from the shifted probability forces it to round down,
and fixes the issue on this clip, but breaks pretty much every other
test clip I have. Here's what I did.

        for (i = 1; i < 257; i++) {
            if (rac->prob[i] != 0)
                rac->prob[i] = (((uint64_t) rac->prob[i] <<
scale_factor) - 1) / cumul_prob;
            scaled_cumul_prob += rac->prob[i];
        }

I'm hoping someone can help here, as I said I'm not sure how to fix this one.


-- 
-Nathan Caldwell