[FFmpeg-devel] [PATCH] fate.sh: Allow overriding what targets to make for running the tests

[Martin Storsjö]

On Thu, 30 Nov 2023, Rémi Denis-Courmont wrote:

> You can already test it properly as things stand, and reporting is trivial, 
> just not to the FATE website. The question is whether this is worth adding to 
> FATE.

More public test coverage is better than less, isn't it?

> In other words, is publishing on the FATE website worth making the tests 
> coverage and/or the build time worse?

By making the test coverage worse, you mean if I'd be doing the full 
testing of many combinations already, and I'd stop doing that in order to 
do this lesser testing instead? If I'd be doing it (I currently don't) I 
guess that would be my concern, not others?

And whether I want to spend my cpu cycles on testing for a public FATE 
config, even if it only tests a limited amount, in addition what tests I 
(hypothetically) run privately, wasting more CPU on building ffmpeg, 
that's also my concern and not others?

> not to mention confusing the existing website users with weirdly 
> incomplete test results.

FWIW, there are a bunch of otherwise weird, limited configs as well, e.g. 
http://fate.ffmpeg.org/history.cgi?slot=x86_64-archlinux-gcc-disableavcodec. 
Although there, the reason for the limited number of tests is clearly 
visible.

>> Again, for SVE, I'd rather have testing with 1 config (the default, which
>> is longer vectors than one usually encounters in HW) rather than none at
>> all. It won't catch every theoretical issue but practically would catch
>> many things at least.
>
> I find that statement very misleading. This is not a question of testing 1 
> config vs 0. It's a question of testing 1 configuration vs all of them(*), and 
> reporting that one vs reporting all of them elsewhere than FATE.ffmpeg.org. 
> Until/unless somebody does the missing integration.

Currently I test 0 of these configurations. I would like to test 1 such 
config, and publish those results on the FATE website. I don't currently 
test any form of "all configs". And if I wanted to make a private setup 
for testing "all configs", I really don't see how it would be mutually 
exclusive with the publicly posted test results from the one config?

>> And in order to actually test BTI, one has to link with a sysroot that
>> also was built with BTI enabled - I currently use a sysroot extracted from
>> fedora for that. (And my tests for it use -Wl,-z,force-bti.)
>
> I can readily believe how much of a PITA that would be to set up. I can also 
> believe that glibc won't allow masking the guarded page bit in mmap()/
> mprotect().
>
> That does not mean you need different builds to test each of the 4 possible 
> combinations (or 3 if you ignore the case of BTI without PAC, which does not 
> exist in real hardware). Once you have that build, you can test it with 
> whichever QEMU CPU settings.

I didn't mean to imply that one would have to do separate builds for all 
of those. I currently don't do any testing with builds with 
-mbranch-protection=standard (and with different sysroots), but I was 
considering adding one such build, with the fedora sysroot - and only test 
one single configuration with it (with QEMU's defaults of all features 
enabled).


So, to spell out your objection in simpler terms. You are firmly against 
anybody posting test results on FATE that only include checkasm but not 
the rest of the tests, because you consider that this can be 
misleading/confusing to people reading the test results - is that right?

Or would such a setup be acceptable to you, if someone would implement a 
way of running the tests (either the full set or only a subset such as 
chckasm) multiple times with different QEMU configurations, with the same 
build of ffmpeg, within the same FATE run?

// Martin