[Ffmpeg-devel] rounding question

[Loren Merritt]

On Fri, 9 Mar 2007, Marc Hoffman wrote:

> Looking at the set of pixel functions which do xy corner averaging I'm 
> confused by the reference math.  There are two functions one with rounding 
> and the other with out.  The one with rounding adds 4 samples and then adds 
> 1/2 and truncates.  The one that is titled no_rnd adds 1/4 and then 
> truncates.  I'm a little confused by the semantics and am not sure why this 
> is.

I don't know where the terminology comes from (maybe some mpeg standard?), 
but I can explain why those are the two algorithms used.
When dealing with rounding of real numbers, the standard mathematical 
convention is to round .5 up. But when dealing with finite-precision 
numbers (such as this case with only 2 bits of fraction), that would add a 
bias towards larger numbers. i.e. if you run motion compensation for 
several frames without coding a residual, the last frame will be lighter 
than the first.
IIRC the standard in banking is to round .5 to the nearest even integer, 
but video codecs wouldn't want to do that since it adds computational 
complexity.
So mpeg4's algorithm alternates. Half the frames round .5 up, half round 
down.
Note that this only applies to how you treat .5, any other fraction is 
definitely closer to one integer than to another, so they always get 
rounded to nearest.

--Loren Merritt