[FFmpeg-devel] [PATCH] fate: add VVC decoder tests

[Kieran Kunhya]

On Sat, 6 Jan 2024 at 02:35, Nuo Mi <nuomi2021 at gmail.com> wrote:

> On Sat, Jan 6, 2024 at 9:13 AM James Almer <jamrial at gmail.com> wrote:
>
> > On 1/5/2024 10:09 PM, Nuo Mi wrote:
> > > On Sat, Jan 6, 2024 at 5:09 AM James Almer <jamrial at gmail.com> wrote:
> > >
> > >> On 1/3/2024 9:19 PM, James Almer wrote:
> > >>> Signed-off-by: James Almer <jamrial at gmail.com>
> > >>> ---
> > >>>    tests/Makefile                                |   1 +
> > >>>    tests/fate/vvc.mak                            |  50 +++++++
> > >>>    tests/ref/fate/vvc-conformance-APSALF_A_2     |  13 ++
> > >>>    tests/ref/fate/vvc-conformance-APSLMCS_D_1    |  37 +++++
> > >>>    tests/ref/fate/vvc-conformance-APSMULT_A_4    |  53 +++++++
> > >>>    tests/ref/fate/vvc-conformance-AUD_A_3        |  35 +++++
> > >>>    tests/ref/fate/vvc-conformance-BUMP_A_2       |  45 ++++++
> > >>>    tests/ref/fate/vvc-conformance-CROP_B_4       | 105 ++++++++++++++
> > >>>    .../fate/vvc-conformance-CodingToolsSets_A_2  |   7 +
> > >>>    tests/ref/fate/vvc-conformance-DCI_A_3        |   7 +
> > >>>    tests/ref/fate/vvc-conformance-HRD_A_3        |  65 +++++++++
> > >>>    tests/ref/fate/vvc-conformance-OPI_B_3        |   6 +
> > >>>    tests/ref/fate/vvc-conformance-PHSH_B_1       |  11 ++
> > >>>    tests/ref/fate/vvc-conformance-POC_A_1        |  25 ++++
> > >>>    tests/ref/fate/vvc-conformance-PPS_B_1        |  69 +++++++++
> > >>>    tests/ref/fate/vvc-conformance-RAP_A_1        |   6 +
> > >>>    tests/ref/fate/vvc-conformance-SAO_A_3        |  65 +++++++++
> > >>>    tests/ref/fate/vvc-conformance-SCALING_A_1    |  69 +++++++++
> > >>>    tests/ref/fate/vvc-conformance-SLICES_A_3     |  30 ++++
> > >>>    tests/ref/fate/vvc-conformance-SPS_B_1        | 133
> > ++++++++++++++++++
> > >>>    tests/ref/fate/vvc-conformance-STILL_B_1      |  10 ++
> > >>>    tests/ref/fate/vvc-conformance-SUBPIC_A_3     |   9 ++
> > >>>    tests/ref/fate/vvc-conformance-TILE_A_2       |  14 ++
> > >>>    tests/ref/fate/vvc-conformance-VPS_A_3        |   6 +
> > >>>    tests/ref/fate/vvc-conformance-WPP_A_3        |  54 +++++++
> > >>>    tests/ref/fate/vvc-conformance-WP_A_3         |  22 +++
> > >>>    tests/ref/fate/vvc-conformance-WRAP_A_4       |  14 ++
> > >>>    27 files changed, 961 insertions(+)
> > >>>    create mode 100644 tests/fate/vvc.mak
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-APSALF_A_2
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-APSLMCS_D_1
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-APSMULT_A_4
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-AUD_A_3
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-BUMP_A_2
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-CROP_B_4
> > >>>    create mode 100644
> > tests/ref/fate/vvc-conformance-CodingToolsSets_A_2
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-DCI_A_3
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-HRD_A_3
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-OPI_B_3
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-PHSH_B_1
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-POC_A_1
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-PPS_B_1
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-RAP_A_1
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-SAO_A_3
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-SCALING_A_1
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-SLICES_A_3
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-SPS_B_1
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-STILL_B_1
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-SUBPIC_A_3
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-TILE_A_2
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-VPS_A_3
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-WPP_A_3
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-WP_A_3
> > >>>    create mode 100644 tests/ref/fate/vvc-conformance-WRAP_A_4
> > >>
> > >> This is using the samples already in the fate suite. I don't know if
> > >> they give good coverage (I think i chose them to cover as much CBS
> > >> methods as possible, not caring about actual decode paths, and not
> even
> > >> that is complete), so I'd like someone more familiar with the
> > >> conformance suite to suggest better samples to upload and use, too.
> > >>
> > >
> > > Here are the clips and their sources:
> > > https://github.com/ffvvc/tests/tree/main
> > > passed v1 is about 194M , and each clip may have different features.
> It's
> > > better to add them all to vvc-conformance, without name change
> > > passed v2 is about 871M , we can pick some smaller clips that cover all
> > bit
> > > depths and color formats.
> >
> > Yeah, it's not so much about the size of the samples than it is about
> > runtime. That many tests would make a fate run take way too much, even
> > after we add assembly.
> > We need to choose a subset that cover as many decoding cases as possible.
> >
> Good point.
> If I disable the asm code on my machine, "make fate-hevc" takes about 1
> minute and 26 seconds. We can use this as the baseline.
> If we remove the largest 5 files in v1, we can decode it in 5 minutes and
> 13 seconds. We can remove more if we find it's still slow.
> From a feature perspective, TREE_A_HHI_3.bit may cover major features in
> TREE_B_HHI_3.bit, TREE_C_HHI_3.bit. It's reasonable to remove them
>
> Largest 5 files:
> TRANS_B_Chipsnmedia_2.bit
> TRANS_A_Chipsnmedia_2.bit
> TREE_B_HHI_3.bit
> LFNST_D_HHI_3.bit
> TREE_C_HHI_3.bit
>

Could we have an option in the FATE config file to do a full run and allow
admins to turn it on at will?
I appreciate there are a lot of embedded devices where running all the VVC
decodes will take forever. But at the same time there are powerful devices
like M1, NUC etc where this isn't a big deal and I'd like to see VVC tested
in full.

Kieran