[FFmpeg-devel] [PATCH] fate: add VVC decoder tests

[Martin Storsjö]

On Sat, 6 Jan 2024, Kieran Kunhya wrote:

> On Sat, 6 Jan 2024 at 02:35, Nuo Mi <nuomi2021 at gmail.com> wrote:
>
>> On Sat, Jan 6, 2024 at 9:13 AM James Almer <jamrial at gmail.com> wrote:
>>
>> > On 1/5/2024 10:09 PM, Nuo Mi wrote:
>> > > On Sat, Jan 6, 2024 at 5:09 AM James Almer <jamrial at gmail.com> wrote:
>> > >
>> > > Here are the clips and their sources:
>> > > https://github.com/ffvvc/tests/tree/main
>> > > passed v1 is about 194M , and each clip may have different features.
>> It's
>> > > better to add them all to vvc-conformance, without name change
>> > > passed v2 is about 871M , we can pick some smaller clips that cover all
>> > bit
>> > > depths and color formats.
>> >
>> > Yeah, it's not so much about the size of the samples than it is about
>> > runtime. That many tests would make a fate run take way too much, even
>> > after we add assembly.
>> > We need to choose a subset that cover as many decoding cases as possible.
>> >
>> Good point.
>> If I disable the asm code on my machine, "make fate-hevc" takes about 1
>> minute and 26 seconds. We can use this as the baseline.
>> If we remove the largest 5 files in v1, we can decode it in 5 minutes and
>> 13 seconds. We can remove more if we find it's still slow.
>> From a feature perspective, TREE_A_HHI_3.bit may cover major features in
>> TREE_B_HHI_3.bit, TREE_C_HHI_3.bit. It's reasonable to remove them
>>
>> Largest 5 files:
>> TRANS_B_Chipsnmedia_2.bit
>> TRANS_A_Chipsnmedia_2.bit
>> TREE_B_HHI_3.bit
>> LFNST_D_HHI_3.bit
>> TREE_C_HHI_3.bit
>>
>
> Could we have an option in the FATE config file to do a full run and allow
> admins to turn it on at will?
> I appreciate there are a lot of embedded devices where running all the VVC
> decodes will take forever. But at the same time there are powerful devices
> like M1, NUC etc where this isn't a big deal and I'd like to see VVC tested
> in full.

We already have an option that is somewhat like this; 
--disable-large-tests, which is documented as "disable tests that use a 
large amount of memory", primarily intended for small SBCs and similar - 
where running fate takes a long time in any case, but one doesn't want it 
to fail due to some few tests doing 8k resolutions and such.

Not sure if that's the right match here, or if we should add another 
option, like --enable-extra-tests, which would be opt-in?

// Martin