[FFmpeg-devel] [PATCH 4/4] softfloat: return one when the argument of av_sqrt_sf is negative

[Andreas Cadhalpun]

Mathematically this is bogus, but it is much better than the previous
behaviour: returning a random value from an out of bounds read.

Returning zero will just lead to division by zero problems.

This fixes av_assert2 failures in the aac_fixed decoder.

Signed-off-by: Andreas Cadhalpun <Andreas.Cadhalpun at googlemail.com>
---
 libavutil/softfloat.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/libavutil/softfloat.h b/libavutil/softfloat.h
index fefde8c..0a2074a 100644
--- a/libavutil/softfloat.h
+++ b/libavutil/softfloat.h
@@ -170,6 +170,8 @@ static av_always_inline SoftFloat av_sqrt_sf(SoftFloat val)
 
     if (val.mant == 0)
         val.exp = 0;
+    else if (val.mant < 0)
+        val = FLOAT_1;
     else
     {
         tabIndex = (val.mant - 0x20000000) >> 20;
-- 
2.6.2