[FFmpeg-devel] [PATCH] dca: add new fate tests based on the dcadec-samples test suite

[James Almer]

On 2/1/2016 7:51 PM, Hendrik Leppkes wrote:
> On Mon, Feb 1, 2016 at 11:29 PM, James Almer <jamrial at gmail.com> wrote:
>> On 2/1/2016 7:01 PM, Hendrik Leppkes wrote:
>>> ---
>>>  tests/fate/dca.mak                            | 72 +++++++++++++++++++++++++++
>>>  tests/ref/fate/dca-xll_51_16_192_768_0        |  1 +
>>>  tests/ref/fate/dca-xll_51_16_192_768_1        |  1 +
>>>  tests/ref/fate/dca-xll_51_16_192_768_1-dmix_2 |  1 +
>>>  tests/ref/fate/dca-xll_51_24_48_768           |  1 +
>>>  tests/ref/fate/dca-xll_51_24_48_none          |  1 +
>>>  tests/ref/fate/dca-xll_71_24_48_768_0         |  1 +
>>>  tests/ref/fate/dca-xll_71_24_48_768_0-dmix_2  |  1 +
>>>  tests/ref/fate/dca-xll_71_24_48_768_0-dmix_6  |  1 +
>>>  tests/ref/fate/dca-xll_71_24_48_768_1         |  1 +
>>>  tests/ref/fate/dca-xll_71_24_48_768_1-dmix_2  |  1 +
>>>  tests/ref/fate/dca-xll_71_24_48_768_1-dmix_6  |  1 +
>>>  tests/ref/fate/dca-xll_71_24_96_768           |  1 +
>>>  tests/ref/fate/dca-xll_x96_51_24_96_1509      |  1 +
>>>  tests/ref/fate/dca-xll_xch_61_24_48_768       |  1 +
>>>  15 files changed, 86 insertions(+)
>>>  create mode 100644 tests/ref/fate/dca-xll_51_16_192_768_0
>>>  create mode 100644 tests/ref/fate/dca-xll_51_16_192_768_1
>>>  create mode 100644 tests/ref/fate/dca-xll_51_16_192_768_1-dmix_2
>>>  create mode 100644 tests/ref/fate/dca-xll_51_24_48_768
>>>  create mode 100644 tests/ref/fate/dca-xll_51_24_48_none
>>>  create mode 100644 tests/ref/fate/dca-xll_71_24_48_768_0
>>>  create mode 100644 tests/ref/fate/dca-xll_71_24_48_768_0-dmix_2
>>>  create mode 100644 tests/ref/fate/dca-xll_71_24_48_768_0-dmix_6
>>>  create mode 100644 tests/ref/fate/dca-xll_71_24_48_768_1
>>>  create mode 100644 tests/ref/fate/dca-xll_71_24_48_768_1-dmix_2
>>>  create mode 100644 tests/ref/fate/dca-xll_71_24_48_768_1-dmix_6
>>>  create mode 100644 tests/ref/fate/dca-xll_71_24_96_768
>>>  create mode 100644 tests/ref/fate/dca-xll_x96_51_24_96_1509
>>>  create mode 100644 tests/ref/fate/dca-xll_xch_61_24_48_768
>>>
>>> diff --git a/tests/fate/dca.mak b/tests/fate/dca.mak
>>> index d8c1117..78d2f33 100644
>>> --- a/tests/fate/dca.mak
>>> +++ b/tests/fate/dca.mak
>>> @@ -1,3 +1,75 @@
>>> +# dcadec test samples
>>> +DCADEC_SUITE_LOSSLESS_16 = xll_51_16_192_768_0        \
>>> +                           xll_51_16_192_768_1        \
>>> +
>>> +DCADEC_SUITE_LOSSLESS_24 = xll_51_24_48_768           \
>>> +                           xll_51_24_48_none          \
>>> +                           xll_71_24_48_768_0         \
>>> +                           xll_71_24_48_768_1         \
>>> +                           xll_71_24_96_768           \
>>> +                           xll_x96_51_24_96_1509      \
>>> +                           xll_xch_61_24_48_768       \
>>> +
>>> +DCADEC_SUITE_LOSSY       = core_51_24_48_768_0        \
>>> +                           core_51_24_48_768_1        \
>>> +                           x96_51_24_96_1509          \
>>> +                           x96_xch_61_24_96_3840      \
>>> +                           x96_xxch_71_24_96_3840     \
>>> +                           xbr_51_24_48_3840          \
>>> +                           xbr_xch_61_24_48_3840      \
>>> +                           xbr_xxch_71_24_48_3840     \
>>> +                           xch_61_24_48_768           \
>>> +                           xxch_71_24_48_2046         \
>>> +
>>> +define FATE_DCADEC_LOSSLESS_SUITE
>>> +FATE_DCADEC_LOSSLESS += fate-dca-$(1)
>>> +fate-dca-$(1): CMD = md5 -i $(TARGET_SAMPLES)/dts/dcadec-suite/$(1).dtshd -f $(2)
>>
>> Unlike the existing xll sample, all these new ones have six frames max, so you
>> could use framecrc or framemd5 instead.
> 
> Suppose that might make debugging easier in the future if something breaks.
> 
>>
>>> +endef
>>> +
>>> +define FATE_DCADEC_LOSSY_SUITE
>>> +FATE_DCADEC_LOSSY += fate-dca-$(1)
>>> +fate-dca-$(1): CMD = pcm -i $(TARGET_SAMPLES)/dts/dcadec-suite/$(1).dtshd
>>> +fate-dca-$(1): CMP = oneoff
>>
>> Should be ok assuming converting to s16 before doing the comparison is acceptable.
>> I say this because the samples are 24bit if decoded using the fixed codepath, and
>> once we create the reference pcm files and add them to the fate suit, we'll be
>> stuck with them.
> 
> The decoder works in float by default, and all other float codecs use
> this kind of test.

mp3 fate tests use f32 mode, for example. Besides, most of those decoders output
16bit audio when not using floats.

> Not sure how much precision there actually is in the dca float signal
> for lossy, but if someone thinks its worse it we could try to use
> tiny_psnr in float or s32 mode (it doesn't do s24)

tiny_psnr doesn't seem to do s32 either, only f32. How hard would it be to add
it, or s24?

> Would require more careful tuning of the test though, as the precision
> on different systems will definitely differ then.

Changing the tests to make them use f32 seems to need a FUZZ of at least 7.

I'd like foo86 to chime in and give his opinion in the matter, but in any case
this is not a blocker so if you think s16 is fine then go ahead.