[FFmpeg-devel] [PATCH] fate: add VVC decoder tests

[James Almer]

On 1/5/2024 10:09 PM, Nuo Mi wrote:
> On Sat, Jan 6, 2024 at 5:09 AM James Almer <jamrial at gmail.com> wrote:
> 
>> On 1/3/2024 9:19 PM, James Almer wrote:
>>> Signed-off-by: James Almer <jamrial at gmail.com>
>>> ---
>>>    tests/Makefile                                |   1 +
>>>    tests/fate/vvc.mak                            |  50 +++++++
>>>    tests/ref/fate/vvc-conformance-APSALF_A_2     |  13 ++
>>>    tests/ref/fate/vvc-conformance-APSLMCS_D_1    |  37 +++++
>>>    tests/ref/fate/vvc-conformance-APSMULT_A_4    |  53 +++++++
>>>    tests/ref/fate/vvc-conformance-AUD_A_3        |  35 +++++
>>>    tests/ref/fate/vvc-conformance-BUMP_A_2       |  45 ++++++
>>>    tests/ref/fate/vvc-conformance-CROP_B_4       | 105 ++++++++++++++
>>>    .../fate/vvc-conformance-CodingToolsSets_A_2  |   7 +
>>>    tests/ref/fate/vvc-conformance-DCI_A_3        |   7 +
>>>    tests/ref/fate/vvc-conformance-HRD_A_3        |  65 +++++++++
>>>    tests/ref/fate/vvc-conformance-OPI_B_3        |   6 +
>>>    tests/ref/fate/vvc-conformance-PHSH_B_1       |  11 ++
>>>    tests/ref/fate/vvc-conformance-POC_A_1        |  25 ++++
>>>    tests/ref/fate/vvc-conformance-PPS_B_1        |  69 +++++++++
>>>    tests/ref/fate/vvc-conformance-RAP_A_1        |   6 +
>>>    tests/ref/fate/vvc-conformance-SAO_A_3        |  65 +++++++++
>>>    tests/ref/fate/vvc-conformance-SCALING_A_1    |  69 +++++++++
>>>    tests/ref/fate/vvc-conformance-SLICES_A_3     |  30 ++++
>>>    tests/ref/fate/vvc-conformance-SPS_B_1        | 133 ++++++++++++++++++
>>>    tests/ref/fate/vvc-conformance-STILL_B_1      |  10 ++
>>>    tests/ref/fate/vvc-conformance-SUBPIC_A_3     |   9 ++
>>>    tests/ref/fate/vvc-conformance-TILE_A_2       |  14 ++
>>>    tests/ref/fate/vvc-conformance-VPS_A_3        |   6 +
>>>    tests/ref/fate/vvc-conformance-WPP_A_3        |  54 +++++++
>>>    tests/ref/fate/vvc-conformance-WP_A_3         |  22 +++
>>>    tests/ref/fate/vvc-conformance-WRAP_A_4       |  14 ++
>>>    27 files changed, 961 insertions(+)
>>>    create mode 100644 tests/fate/vvc.mak
>>>    create mode 100644 tests/ref/fate/vvc-conformance-APSALF_A_2
>>>    create mode 100644 tests/ref/fate/vvc-conformance-APSLMCS_D_1
>>>    create mode 100644 tests/ref/fate/vvc-conformance-APSMULT_A_4
>>>    create mode 100644 tests/ref/fate/vvc-conformance-AUD_A_3
>>>    create mode 100644 tests/ref/fate/vvc-conformance-BUMP_A_2
>>>    create mode 100644 tests/ref/fate/vvc-conformance-CROP_B_4
>>>    create mode 100644 tests/ref/fate/vvc-conformance-CodingToolsSets_A_2
>>>    create mode 100644 tests/ref/fate/vvc-conformance-DCI_A_3
>>>    create mode 100644 tests/ref/fate/vvc-conformance-HRD_A_3
>>>    create mode 100644 tests/ref/fate/vvc-conformance-OPI_B_3
>>>    create mode 100644 tests/ref/fate/vvc-conformance-PHSH_B_1
>>>    create mode 100644 tests/ref/fate/vvc-conformance-POC_A_1
>>>    create mode 100644 tests/ref/fate/vvc-conformance-PPS_B_1
>>>    create mode 100644 tests/ref/fate/vvc-conformance-RAP_A_1
>>>    create mode 100644 tests/ref/fate/vvc-conformance-SAO_A_3
>>>    create mode 100644 tests/ref/fate/vvc-conformance-SCALING_A_1
>>>    create mode 100644 tests/ref/fate/vvc-conformance-SLICES_A_3
>>>    create mode 100644 tests/ref/fate/vvc-conformance-SPS_B_1
>>>    create mode 100644 tests/ref/fate/vvc-conformance-STILL_B_1
>>>    create mode 100644 tests/ref/fate/vvc-conformance-SUBPIC_A_3
>>>    create mode 100644 tests/ref/fate/vvc-conformance-TILE_A_2
>>>    create mode 100644 tests/ref/fate/vvc-conformance-VPS_A_3
>>>    create mode 100644 tests/ref/fate/vvc-conformance-WPP_A_3
>>>    create mode 100644 tests/ref/fate/vvc-conformance-WP_A_3
>>>    create mode 100644 tests/ref/fate/vvc-conformance-WRAP_A_4
>>
>> This is using the samples already in the fate suite. I don't know if
>> they give good coverage (I think i chose them to cover as much CBS
>> methods as possible, not caring about actual decode paths, and not even
>> that is complete), so I'd like someone more familiar with the
>> conformance suite to suggest better samples to upload and use, too.
>>
> 
> Here are the clips and their sources:
> https://github.com/ffvvc/tests/tree/main
> passed v1 is about 194M , and each clip may have different features. It's
> better to add them all to vvc-conformance, without name change
> passed v2 is about 871M , we can pick some smaller clips that cover all bit
> depths and color formats.

Yeah, it's not so much about the size of the samples than it is about 
runtime. That many tests would make a fate run take way too much, even 
after we add assembly.
We need to choose a subset that cover as many decoding cases as possible.