[FFmpeg-devel] [PATCH] libavutil/softfloat: Fix normalizations.

[Michael Niedermayer]

On Fri, Jun 05, 2015 at 09:27:56AM +0000, Nedeljko Babic wrote:
> >> >> -        while((a.mant + 0x20000000U)<0x40000000U){
> >> >> +        while((FFABS(a.mant) + 0x20000000U)<0x40000000U){
> >> >
> >> >when exactly is this needed and how does a.mant reach that value
> >> >for which this is needed?
> >> >FFABS would significantly slow this down i suspect and it just looks
> >> >wrong
> >> >
> >>
> >> The problem occurs for some negative numbers during normalization.
> >>
> >> For example, please consider if 0xFFFFFFD80 is sent to av_normalize_sf.
> >
> >considering that this is not a 32bit value and mant is, theres no
> >way that can happen
> >
> >
> >>
> >> The while loop has a condition: (a.mant + 0x20000000U)<0x40000000U.
> >> 0xFFFFD80 is cast to unsigned and we have:
> >
> >and that lost its highest 8bit
> >also FFABS() does not chaneg the result for this
> 
> Ok, this was relay bad example (and explanation)... Sorry.
> 
> Lets try another example:
> 
> int32_t a = 0xFFFFFF00;
> SoftFloat sf = av_int2sf(a, xxx); // xxx is not important here
> 
> If original condition in normalization is used we will have:
> sf.mant = 0xc0000000
> 
> If FFABS is used in condition, there will be one iteration less and we will have:
> sf.mant = 0xe0000000

fixed

thanks

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Opposition brings concord. Out of discord comes the fairest harmony.
-- Heraclitus
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20150605/6fae16ed/attachment.asc>