[FFmpeg-devel] [PATCH] fate.sh: Allow overriding what targets to make for running the tests

[Rémi Denis-Courmont]

Le 30 novembre 2023 23:13:59 GMT+02:00, "Martin Storsjö" <martin at martin.st> a écrit :
>On Thu, 30 Nov 2023, Rémi Denis-Courmont wrote:
>
>> You can already test it properly as things stand, and reporting is trivial, just not to the FATE website. The question is whether this is worth adding to FATE.
>
>More public test coverage is better than less, isn't it?

That's a false dichotomy.

>> In other words, is publishing on the FATE website worth making the tests coverage and/or the build time worse?
>
>By making the test coverage worse, you mean if I'd be doing the full testing of many combinations already, and I'd stop doing that in order to do this lesser testing instead? If I'd be doing it (I currently don't) I guess that would be my concern, not others?

No. The point is that this is adding a small hack that works for one specific case for a short while (testing Armv8 IMM8 and DP), but is known not to be sufficient anyway (for SVE, PAuth, RVV, etc).

In the end, it's all about not adding inadequate interfaces and supporting/publishing bad solutions. It's certainly not as bad as if it were a public C API, but that doesn't make it good. Normally "insufficient" interfaces don't get merged for a variety of reasons.

>>> Again, for SVE, I'd rather have testing with 1 config (the default, which
>>> is longer vectors than one usually encounters in HW) rather than none at
>>> all. It won't catch every theoretical issue but practically would catch
>>> many things at least.
>> 
>> I find that statement very misleading. This is not a question of testing 1 config vs 0. It's a question of testing 1 configuration vs all of them(*), and reporting that one vs reporting all of them elsewhere than FATE.ffmpeg.org. Until/unless somebody does the missing integration.
>
>Currently I test 0 of these configurations. I would like to test 1 such config, and publish those results on the FATE website. I don't currently test any form of "all configs". And if I wanted to make a private setup for testing "all configs", I really don't see how it would be mutually exclusive with the publicly posted test results from the one config?
>
>>> And in order to actually test BTI, one has to link with a sysroot that
>>> also was built with BTI enabled - I currently use a sysroot extracted from
>>> fedora for that. (And my tests for it use -Wl,-z,force-bti.)
>> 
>> I can readily believe how much of a PITA that would be to set up. I can also believe that glibc won't allow masking the guarded page bit in mmap()/
>> mprotect().
>> 
>> That does not mean you need different builds to test each of the 4 possible combinations (or 3 if you ignore the case of BTI without PAC, which does not exist in real hardware). Once you have that build, you can test it with whichever QEMU CPU settings.
>
>I didn't mean to imply that one would have to do separate builds for all of those. I currently don't do any testing with builds with -mbranch-protection=standard (and with different sysroots), but I was considering adding one such build, with the fedora sysroot - and only test one single configuration with it (with QEMU's defaults of all features enabled).
>
>
>So, to spell out your objection in simpler terms. You are firmly against anybody posting test results on FATE that only include checkasm but not the rest of the tests, because you consider that this can be misleading/confusing to people reading the test results - is that right?
>
>Or would such a setup be acceptable to you, if someone would implement a way of running the tests (either the full set or only a subset such as chckasm) multiple times with different QEMU configurations, with the same build of ffmpeg, within the same FATE run?
>
>// Martin
>_______________________________________________
>ffmpeg-devel mailing list
>ffmpeg-devel at ffmpeg.org
>https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
>To unsubscribe, visit link above, or email
>ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".