[FFmpeg-devel] [PATCH 06/11] avutil/half2float: adjust conversion of NaN

[Andreas Rheinhardt]

Timo Rothenpieler:
> On 10.08.2022 23:24, Andreas Rheinhardt wrote:
>> Timo Rothenpieler:
>>> IEEE-754 differentiates two different kind of NaNs.
>>> Quiet and Signaling ones. They are differentiated by the MSB of the
>>> mantissa.
>>>
>>> For whatever reason, actual hardware conversion of half to single always
>>> sets the signaling bit to 1 if the mantissa is != 0, and to 0 if it's 0.
>>> So our code has to follow suite or fate-testing hardware float16 will be
>>> impossible.
>>
>> What does the exr spec say about quiet and signaling nans?
> 
> Not sure how exr would be involved here.

Your patch changes the output of an exr-test. The output of the exr
decoder is presumably determined by the exr spec. There is after all the
possibility that what hardware does in hardware and what this patch does
in software is incompatible with what exr specifies.

> But I tested this on both aarch64, x86 with sse2 emulation and x86 f16c
> on alderlake and zen2.
> They all perfectly agree and match 100% what this changed code produces
> for the entire range of 65k possible values.
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".