[FFmpeg-devel] [PATCH] fate.sh: Allow overriding what targets to make for running the tests

[Martin Storsjö]

On Thu, 30 Nov 2023, Rémi Denis-Courmont wrote:

> Le 27 novembre 2023 23:55:18 GMT+02:00, "Martin Storsjö" <martin at martin.st> a écrit :
>> On Mon, 27 Nov 2023, Rémi Denis-Courmont wrote:
>>
>>> Le maanantaina 27. marraskuuta 2023, 14.31.18 EET Martin Storsjö a écrit :
>>>> This can be useful if doing testing of uncommon CPU extensions by
>>>> running tests with QEMU (by configuring with e.g.
>>>> "target_exec=qemu-aarch64"), by only running the checkasm tests,
>>>> to get a reasonable test coverage without excessive test runtime.
>>> 
>>> For the purpose of testing future or bleeding-edge CPU extensions on emulator, you would normally want to be able to actually filter those in. That is more of a matter of patching checkasm than FATE.
>>
>> Sorry, can you elaborate on what you mean with "filter those in" here?
>
> You're running all checkasm tests, not just those that require the 
> emulator.
>
> But what's potentially much worse is that you're triggering a whole 
> build, or it's not entirely clear from the description how you'd reuse 
> an existing build.

Yeah, I wouldn't reuse an existing build here. For the setup I have in 
mind, one build doesn't take too horribly long (either on an old desktop 
x86 machine, or a moderate aarch64 server) - so it's not ideal but not a 
dealbreaker anyway (while running all of fate with qemu takes one 
magnitude longer).

For the other setup I intended to test, to test AArch64 PAC and BTI, I 
would do a separate build with -mbranch-protection=standard anyway.

> For Armv8, that's just bad. For RV, that's terrible, as we need to run 
> the same checkasm with different emulator configuration (different 
> $QEMU_CPU in the case of QEMU): one per vector length. Armv9 will 
> potentially have the same problem if FFmpeg grows SVE(2) support.

Yes, for SVE I would ideally like to test all vector lengths (I did such a 
setup for x264 recently, when someone was proposing some SVE codepaths). I 
don't have a neat idea for how to integrate that into FATE, and this patch 
doesn't buy us that indeed.

But running tests with the default QEMU settings would at least test with 
a larger vector length than the usual, so it would provide at least some 
coverage. Not exhaustive, but at least something.

So the setup I have in mind wouldn't cover all those cases, but it would 
at least fix some current gaps in testing coverage. I guess it's a case of 
whether we should let perfect be the enemy of good; adding this lets us 
easily add a fair bit of more coverage, in particular of the (new) 
handwritten asm. And it shouldn't get in the way of doing other better 
solutions at a later point.

// Martin