[FFmpeg-devel] [PATCH] fate.sh: Allow overriding what targets to make for running the tests

[Rémi Denis-Courmont]

Le torstaina 30. marraskuuta 2023, 18.28.39 EET Martin Storsjö a écrit :
> On Thu, 30 Nov 2023, Rémi Denis-Courmont wrote:
> > Le torstaina 30. marraskuuta 2023, 17.34.31 EET Martin Storsjö a écrit :
> >> Yeah, I wouldn't reuse an existing build here. For the setup I have in
> >> mind, one build doesn't take too horribly long (either on an old desktop
> >> x86 machine, or a moderate aarch64 server) - so it's not ideal but not a
> >> dealbreaker anyway (while running all of fate with qemu takes one
> >> magnitude longer).
> > 
> > Well it's pretty much a deal breaker for Armv9 and RV. I can understand
> > wanting to build on a comfy x86 server, but doing different builds just to
> > change QEMU CPU flags is IMO inept.
> 
> Yes. But for doing one single run with QEMU, I don't mind.

You can already test it properly as things stand, and reporting is trivial, 
just not to the FATE website. The question is whether this is worth adding to 
FATE. In other words, is publishing on the FATE website worth making the tests 
coverage and/or the build time worse? not to mention confusing the existing 
website users with weirdly incomplete test results.

> Again, for SVE, I'd rather have testing with 1 config (the default, which
> is longer vectors than one usually encounters in HW) rather than none at
> all. It won't catch every theoretical issue but practically would catch
> many things at least.

I find that statement very misleading. This is not a question of testing 1 
config vs 0. It's a question of testing 1 configuration vs all of them(*), and 
reporting that one vs reporting all of them elsewhere than FATE.ffmpeg.org. 
Until/unless somebody does the missing integration.

(*) at least those that QEMU supports

> Are you volunteering to write FATE integration to run checkasm multiple 
> times with different QEMU settings, so I can wait for that instead of 
> having much improved public test coverage right now?

Of course I will not volunteer, given that the RISE project already has an 
outstanding RfP which will likely require this done professionally:
https://hubs.la/Q029hwpS0
(That does not mean that I would have volunteered otherwise, just that the 
question is moot as far as I am concerned and for the time being.)

> > Sure, we could just build once and run several times checkasm with a
> > separate script, as I already pointed out. But then this patch is
> > completely unnecessary.
> 
> Indeed, that's trivial to do for a private testing setup.
> 
> >> For the other setup I intended to test, to test AArch64 PAC and BTI, I
> >> would do a separate build with -mbranch-protection=standard anyway.
> > 
> > That does not make much sense to me. PAC and BTI should be enabled by
> > default in compatibility mode (for ARMv8.0-8.2 builds) or
> > noncompatibility mode (for ARMv8.3+ builds).
> 
> Maybe it should - but it currently isn't.

That's really up to whoever set up the AArch64 builds to fix their build flags 
TBH (I believe that the assembler is already sorted). And at least for PAuth, 
that should be sufficient, as support from the C runtime is not required.

> And in order to actually test BTI, one has to link with a sysroot that
> also was built with BTI enabled - I currently use a sysroot extracted from
> fedora for that. (And my tests for it use -Wl,-z,force-bti.)

I can readily believe how much of a PITA that would be to set up. I can also 
believe that glibc won't allow masking the guarded page bit in mmap()/
mprotect().

That does not mean you need different builds to test each of the 4 possible 
combinations (or 3 if you ignore the case of BTI without PAC, which does not 
exist in real hardware). Once you have that build, you can test it with 
whichever QEMU CPU settings. Surely Fedora, of all distros, is not going to 
treat Armv8.5-BTI as a distinct arch from AArch64 whilst Arm made sure it was 
both backward-compatible and runtime-tunable.

-- 
レミ・デニ-クールモン
http://www.remlab.net/