[FFmpeg-devel] [PATCH] libavutil/softfloat: Fix normalizations.

[Nedeljko Babic]

>> >> -        while((a.mant + 0x20000000U)<0x40000000U){
>> >> +        while((FFABS(a.mant) + 0x20000000U)<0x40000000U){
>> >
>> >when exactly is this needed and how does a.mant reach that value
>> >for which this is needed?
>> >FFABS would significantly slow this down i suspect and it just looks
>> >wrong
>> >
>>
>> The problem occurs for some negative numbers during normalization.
>>
>> For example, please consider if 0xFFFFFFD80 is sent to av_normalize_sf.
>
>considering that this is not a 32bit value and mant is, theres no
>way that can happen
>
>
>>
>> The while loop has a condition: (a.mant + 0x20000000U)<0x40000000U.
>> 0xFFFFD80 is cast to unsigned and we have:
>
>and that lost its highest 8bit
>also FFABS() does not chaneg the result for this

Ok, this was relay bad example (and explanation)... Sorry.

Lets try another example:

int32_t a = 0xFFFFFF00;
SoftFloat sf = av_int2sf(a, xxx); // xxx is not important here

If original condition in normalization is used we will have:
sf.mant = 0xc0000000

If FFABS is used in condition, there will be one iteration less and we will have:
sf.mant = 0xe0000000

This is due to an overflow (caused by casting negative values to unsigned) when FFABS is not used.

-Nedeljko