[FFmpeg-devel] [PATCH] correct make test failure from 15261 release until now (15899)

[Vitor Sessak]

Michael Niedermayer wrote:
> On Fri, Nov 21, 2008 at 11:23:57PM +0100, Vitor Sessak wrote:
>> Michael Niedermayer wrote:
>>> On Fri, Nov 21, 2008 at 09:15:30PM +0100, Vitor Sessak wrote:
>>>> David Geldreich wrote:
>>>>> Hello Guillaume,
>>>>>
>>>>> Le 21 nov. 08 ? 17:25, Guillaume POIRIER a ?crit :
>>>>>
>>>>>> This is not the way to go. Reg tests pass on AMD64/Linux, so the code
>>>>>> must be fixed to work the same on any plateform. The md5sum should not
>>>>>> match X plateform results, but all plateforms result.
>>>>> That's why I made another post to tell to ignore my proposed patch.
>>>>>
>>>>> I found no way of making sin/sinf works the same way on all the  
>>>>> platform. In my case, OSX ppc and intel gives different results.
>>>>>
>>>>> So changes r15261 and r14982 are incomplete ... they correct the  
>>>>> problem for AMD64 but breaks in on Intel32.
>>>>>
>>>>> We must iterate to find a "stable" sine window generating function.
>>>> Even if we find a way to generate a sine window in an arch-independent 
>>>> way, the codec still uses floating points in other places, so if it ever 
>>>> is bit-identical across PPC and I32, I don't see any reason not to see a 
>>>> different output when testing on ARM or SH or GCC 6.4 or whatever we'll 
>>>> encounter in future. Unless someone tells me why it is supposed to work 
>>>> as is, I think that this test should be removed...
>>> ratecontrol in video uses floats, and other parts do too, we arent seeing
>>> problems with these and arent disabling them
>>>
>>> If you argue that the wma test should be disabled because it is not
>>> matching between some important systems, thats something i can understand
>>>
>>> but, arguing that itz should be disabled because it might theoretically not
>>> work on some architecture or not yet existing compiler is well ...
>> Ok, I didn't made myself clear. It is not just that it might 
>> theoretically not work, but I see no reason it is more likely to work 
>> than not in a completely valid system, and I don't find this acceptable. 
> 
> And thats where i cant follow your "logic".
> What you suggest is strictly speaking to disable a test because it may or
> lets even say it likely would not work in some "future" case.
> 
> Regression tests exist primarely to allow developers to test their changes,
> find issues early, find out which part exactly broke, ...
> 
> Now if you drop a test you totally loose the protection this test has
> provided.
> 
> But let me use a even more extreem case to make clear what i mean
> lets assume every single regression test failes on every single system
> except x86 and ppc.
> the 99% of development that is done is done on ppc & x86, it would use
> and benefit from the test like always.
> the <1% of development done on other platforms would not be able to use
> the regression tests
> 
> Now compare this to your suggestion of removing such tests because
> they fail on some system
> 99% of the development suddenly looses the functionality of the regression
> tests
> 1% gains nothing, broken tests or no tests is pretty much the same.
> 
> And the problematic regression tests surely could be split out, but
> disabling them for the 99% of developers for whom they work fine
> is not a good idea IMHO

This I think is a good idea, to keep the test but only enable it for the 
compilers/archs where it is supposed to work. It allows to keep testing 
for breakages but will avoid that porters lose time tracking the 
non-existing bugs that are making regtests to fail. As it is today, the 
porters will think that his build of ffmpeg is producing broken output 
when it is not the case.

-Vitor