[FFmpeg-devel] [PATCH] checkasm/af_afir: relax the max allowed absolute difference

[James Almer]

On 1/13/2019 11:32 AM, Derek Buitenhuis wrote:
> On 13/01/2019 02:44, James Almer wrote:
>> Michael suggested 1000*FLT_EPSILON but IMO that's too big and may hide
>> errors in future implementations.
>> The value i used is the smallest value i found that didn't fail after
>> several runs. 6.1e-05 for example fails.
> 
> I figured that's how it was chosen (though not documented). I thought there
> be a way to calculate it properly instead of empirically.

If there is, i don't know it.

Float based fate tests have been fine tuned before when different
architectures proved a certain stddev value was not lax enough, so this
one could be increased if needed as well, but if you prefer i can use a
big enough multiple of FLT_EPSILON instead.