[FFmpeg-devel] [PATCH] fate: Allow running multiple rounds of tests with differing settings

[Martin Storsjö]

On Wed, 13 Dec 2023, Rémi Denis-Courmont wrote:

> Le tiistaina 12. joulukuuta 2023, 0.14.06 EET Martin Storsjö a écrit :
>> This can be used to run tests multple times, with e.g. differing
>> QEMU settings, by adding something like this to the FATE configuration
>> file:
>>
>>     target_exec="qemu-aarch64-static"
>>     fate_targets="fate-checkasm fate-cpu"
>>
>>     fate_environments="sve128 sve256 sve512"
>>     sve128_env="QEMU_CPU=max,sve128=on"
>>     sve256_env="QEMU_CPU=max,sve256=on"
>>     sve512_env="QEMU_CPU=max,sve512=on"
>
> I'm fine with that, but for the sake of generality, shouldn't rather the 
> entire target_exec prefix be indirected? Some runners may want to use 
> command line flags rather than environment variables.

Yes - that's also doable. One can e.g. do this:

---
target_exec="qemu-aarch64-static -cpu \$(MY_CPU)"
fate_targets="fate-checkasm fate-cpu"

fate_environments="sve128 sve256 sve512"
sve128_env="MY_CPU=max,sve128=on"
sve256_env="MY_CPU=max,sve256=on"
sve512_env="MY_CPU=max,sve512=on"
---

That way, one can also make the whole target_exec be e.g. \$(EXEC_CMD) and 
set the full command via the individual envs. It's not quite as 
comfortable, but should be doable and allow fully generic setups.

(One could also consider just doing feat1_env="TARGET_EXEC=..." and not 
setting target_exec in the fate config at all - however that's probably 
not a good option and it has one surprising gotcha. The makefile level 
TARGET_EXEC variable needs to have a trailing space when it is set to a 
nonempty value, and injecting that might be annoying or ugly.)

// Martin