[FFmpeg-devel] Backporting makefile targets fate-list-failing and fate-clear-reports

[Alexander Strasser]

Hi all!

I'm thinking about backporting commit 1e76bd2f394a01c1 [1] to all releases
listed on our download page.

It is a new feature for the build system, but not used to build ffmpeg
or otherwise related to the generated binaries.

It's of utility when doing backports because it makes it easy to run
all fate tests with -k and to get a list of the failed tests after.

Are there any objections to backporting this convenience fate feature?

Best regards,
  Alexander

1. commit 1e76bd2f394a01c19073160c380adbcaa779f474 on master:
```
Author: Martin Storsjö <martin at martin.st>
Date:   Thu Nov 7 11:31:40 2024 +0200

    fate: Add a target for listing failed tests

    If running tests with "make -j<N> fate", the execution will stop
    after the first failing test. To get an overview of the whole
    test suite, one rather would run "make -k -j<N> fate", which then
    again buries the results about what tests actually failed further
    up in the console log.

    Add a target so one can run "make fate-list-failing", to see a list
    of all tests that failed the last time they were executed.

    Also add a companion target "fate-clear-reports" which removes all
    the old test reports. (When executing a subset of tests, the report
    files of all tests that aren't executed stay untouched. This also
    allows getting rid of reports for tests that no longer are present
    in the testsuite.)

    Co-authored-by: Alexander Strasser <eclipse7 at gmx.net>

    Signed-off-by: Martin Storsjö <martin at martin.st>

diff --git a/doc/build_system.txt b/doc/build_system.txt
index 0b1b0c2054..053d449862 100644
--- a/doc/build_system.txt
+++ b/doc/build_system.txt
@@ -30,6 +30,13 @@ fate
 fate-list
     List all fate/regression test targets.

+fate-list-failing
+    List the fate tests that failed the last time they were executed.
+
+fate-clear-reports
+    Remove the test reports from previous test executions (getting rid of
+    potentially stale results from fate-list-failing).
+
 install
     Install headers, libraries and programs.

diff --git a/doc/fate.texi b/doc/fate.texi
index 17644ce65a..7a2e0edfcc 100644
--- a/doc/fate.texi
+++ b/doc/fate.texi
@@ -208,6 +208,13 @@ Download/synchronize sample files to the configured samples directory.
 @item fate-list
 Will list all fate/regression test targets.

+ at item fate-list-failing
+List the fate tests that failed the last time they were executed.
+
+ at item fate-clear-reports
+Remove the test reports from previous test executions (getting rid of
+potentially stale results from fate-list-failing).
+
 @item fate
 Run the FATE test suite (requires the fate-suite dataset).
 @end table
diff --git a/tests/Makefile b/tests/Makefile
index 9b70145015..f9f5fc07f3 100644
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -313,6 +313,12 @@ $(FATE): $(FATE_UTILS:%=tests/%$(HOSTEXESUF)) | $(FATE_OUTDIRS)
 fate-list:
        @printf '%s\n' $(sort $(FATE))

+fate-list-failing:
+       @! ls tests/data/fate/*.rep >/dev/null 2>&1 || awk -F: '$$2 != 0 { print "fate-" $$1 }' tests/data/fate/*.rep
+
+fate-clear-reports:
+       @rm -f tests/data/fate/*.rep
+
 coverage.info: TAG = LCOV
 coverage.info:
        $(M)lcov -q -d $(CURDIR) -b $(patsubst src%,./,$(SRC_LINK)) --capture | \
```