[FFmpeg-devel] [PATCH] dca: add new fate tests based on the dcadec-samples test suite

[Hendrik Leppkes]

On Tue, Feb 2, 2016 at 3:05 AM, James Almer <jamrial at gmail.com> wrote:
> On 2/1/2016 7:51 PM, Hendrik Leppkes wrote:
>> On Mon, Feb 1, 2016 at 11:29 PM, James Almer <jamrial at gmail.com> wrote:
>>> On 2/1/2016 7:01 PM, Hendrik Leppkes wrote:
>>>> ---
>>>>  tests/fate/dca.mak                            | 72 +++++++++++++++++++++++++++
>>>>  tests/ref/fate/dca-xll_51_16_192_768_0        |  1 +
>>>>  tests/ref/fate/dca-xll_51_16_192_768_1        |  1 +
>>>>  tests/ref/fate/dca-xll_51_16_192_768_1-dmix_2 |  1 +
>>>>  tests/ref/fate/dca-xll_51_24_48_768           |  1 +
>>>>  tests/ref/fate/dca-xll_51_24_48_none          |  1 +
>>>>  tests/ref/fate/dca-xll_71_24_48_768_0         |  1 +
>>>>  tests/ref/fate/dca-xll_71_24_48_768_0-dmix_2  |  1 +
>>>>  tests/ref/fate/dca-xll_71_24_48_768_0-dmix_6  |  1 +
>>>>  tests/ref/fate/dca-xll_71_24_48_768_1         |  1 +
>>>>  tests/ref/fate/dca-xll_71_24_48_768_1-dmix_2  |  1 +
>>>>  tests/ref/fate/dca-xll_71_24_48_768_1-dmix_6  |  1 +
>>>>  tests/ref/fate/dca-xll_71_24_96_768           |  1 +
>>>>  tests/ref/fate/dca-xll_x96_51_24_96_1509      |  1 +
>>>>  tests/ref/fate/dca-xll_xch_61_24_48_768       |  1 +
>>>>  15 files changed, 86 insertions(+)
>>>>  create mode 100644 tests/ref/fate/dca-xll_51_16_192_768_0
>>>>  create mode 100644 tests/ref/fate/dca-xll_51_16_192_768_1
>>>>  create mode 100644 tests/ref/fate/dca-xll_51_16_192_768_1-dmix_2
>>>>  create mode 100644 tests/ref/fate/dca-xll_51_24_48_768
>>>>  create mode 100644 tests/ref/fate/dca-xll_51_24_48_none
>>>>  create mode 100644 tests/ref/fate/dca-xll_71_24_48_768_0
>>>>  create mode 100644 tests/ref/fate/dca-xll_71_24_48_768_0-dmix_2
>>>>  create mode 100644 tests/ref/fate/dca-xll_71_24_48_768_0-dmix_6
>>>>  create mode 100644 tests/ref/fate/dca-xll_71_24_48_768_1
>>>>  create mode 100644 tests/ref/fate/dca-xll_71_24_48_768_1-dmix_2
>>>>  create mode 100644 tests/ref/fate/dca-xll_71_24_48_768_1-dmix_6
>>>>  create mode 100644 tests/ref/fate/dca-xll_71_24_96_768
>>>>  create mode 100644 tests/ref/fate/dca-xll_x96_51_24_96_1509
>>>>  create mode 100644 tests/ref/fate/dca-xll_xch_61_24_48_768
>>>>
>>>> diff --git a/tests/fate/dca.mak b/tests/fate/dca.mak
>>>> index d8c1117..78d2f33 100644
>>>> --- a/tests/fate/dca.mak
>>>> +++ b/tests/fate/dca.mak
>>>> @@ -1,3 +1,75 @@
>>>> +# dcadec test samples
>>>> +DCADEC_SUITE_LOSSLESS_16 = xll_51_16_192_768_0        \
>>>> +                           xll_51_16_192_768_1        \
>>>> +
>>>> +DCADEC_SUITE_LOSSLESS_24 = xll_51_24_48_768           \
>>>> +                           xll_51_24_48_none          \
>>>> +                           xll_71_24_48_768_0         \
>>>> +                           xll_71_24_48_768_1         \
>>>> +                           xll_71_24_96_768           \
>>>> +                           xll_x96_51_24_96_1509      \
>>>> +                           xll_xch_61_24_48_768       \
>>>> +
>>>> +DCADEC_SUITE_LOSSY       = core_51_24_48_768_0        \
>>>> +                           core_51_24_48_768_1        \
>>>> +                           x96_51_24_96_1509          \
>>>> +                           x96_xch_61_24_96_3840      \
>>>> +                           x96_xxch_71_24_96_3840     \
>>>> +                           xbr_51_24_48_3840          \
>>>> +                           xbr_xch_61_24_48_3840      \
>>>> +                           xbr_xxch_71_24_48_3840     \
>>>> +                           xch_61_24_48_768           \
>>>> +                           xxch_71_24_48_2046         \
>>>> +
>>>> +define FATE_DCADEC_LOSSLESS_SUITE
>>>> +FATE_DCADEC_LOSSLESS += fate-dca-$(1)
>>>> +fate-dca-$(1): CMD = md5 -i $(TARGET_SAMPLES)/dts/dcadec-suite/$(1).dtshd -f $(2)
>>>
>>> Unlike the existing xll sample, all these new ones have six frames max, so you
>>> could use framecrc or framemd5 instead.
>>
>> Suppose that might make debugging easier in the future if something breaks.
>>
>>>
>>>> +endef
>>>> +
>>>> +define FATE_DCADEC_LOSSY_SUITE
>>>> +FATE_DCADEC_LOSSY += fate-dca-$(1)
>>>> +fate-dca-$(1): CMD = pcm -i $(TARGET_SAMPLES)/dts/dcadec-suite/$(1).dtshd
>>>> +fate-dca-$(1): CMP = oneoff
>>>
>>> Should be ok assuming converting to s16 before doing the comparison is acceptable.
>>> I say this because the samples are 24bit if decoded using the fixed codepath, and
>>> once we create the reference pcm files and add them to the fate suit, we'll be
>>> stuck with them.
>>
>> The decoder works in float by default, and all other float codecs use
>> this kind of test.
>
> mp3 fate tests use f32 mode, for example. Besides, most of those decoders output
> 16bit audio when not using floats.
>
>> Not sure how much precision there actually is in the dca float signal
>> for lossy, but if someone thinks its worse it we could try to use
>> tiny_psnr in float or s32 mode (it doesn't do s24)
>
> tiny_psnr doesn't seem to do s32 either, only f32. How hard would it be to add
> it, or s24?

If anything I think we should make it test f32 since thats the native
output mode, or stick to the usual s16 as converted by swr.
Using the bitexact fixed point code path for those samples to get s24
out of the decoder would leave the float path entirely untested.

>
>> Would require more careful tuning of the test though, as the precision
>> on different systems will definitely differ then.
>
> Changing the tests to make them use f32 seems to need a FUZZ of at least 7.

mp3 uses a fuzz of 17, so its likely going to require higher fuzz
values on different systems.
Will only find out after seeing it fail on FATE on a bunch of systems though.

>
> I'd like foo86 to chime in and give his opinion in the matter, but in any case
> this is not a blocker so if you think s16 is fine then go ahead.

We can also go with f32, its not going to be "worse" than a s16 test,
just need to be prepared to update the fuzz values in the first couple
days.

- Hendrik