[FFmpeg-devel] [PATCH] fate/jpeg2000dec: add missing ISO/IEC 15444-4 conformance tests

WATANABE Osamu owatanab at es.takushoku-u.ac.jp
Mon Nov 11 10:19:31 EET 2024 

I have confirmed that these failures in FATE were due to the insufficient floating point precision of a 32-bit environment.

The commit 82467b635efced67c1767cb810af1f3c31a2e493 introduces the improved dequantization in FF_DWT97_INT path and makes FFmpeg compliant for ISO/IEC 15444-4 (Conformance testing.)

Actually, both 64 and 32-bit FFmpeg with the commit pass the conformance testing.

Current FATE tests for JPEG 2000 are to check the CRC of reconstructed images.
Checking CRC is unsuitable for lossy codestream because even a patch that can improve the quality of a reconstructed image cannot pass the FATE. Passing FATE is, of course, important, but passing the official (ISO's) conformance tests is essential, too.

I recommend introducing a way to allow some tolerance in FATE tests for JPEG 2000 lossy codestreams.
For example, including the original files and using the PSNR value in FATE will be an option.
 

> On Nov 10, 2024, at 10:40, Michael Niedermayer <michael at niedermayer.cc> wrote:
> 
> On Fri, Nov 08, 2024 at 01:09:34PM -0800, pal at sandflow.com wrote:
>> From: Pierre-Anthony Lemieux <pal at palemieux.com>
>> 
>> ---
>> tests/fate/jpeg2000.mak                  | 122 ++++++++++++++++++++++-
>> tests/ref/fate/jpeg2000dec-ds0_hm_15_b8  |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_02_b11 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_02_b12 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_03_b11 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_03_b14 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_04_b11 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_04_b12 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_05_b11 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_05_b12 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_07_b11 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_07_b15 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_07_b16 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_08_b11 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_08_b15 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_08_b16 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_09_b11 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_10_b11 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_11_b10 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_12_b11 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_14_b11 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_15_b11 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_15_b14 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds0_ht_16_b11 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds1_ht_01_b11 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds1_ht_01_b12 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds1_ht_02_b11 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds1_ht_02_b12 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds1_ht_03_b11 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds1_ht_03_b12 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds1_ht_04_b9  |   6 ++
>> tests/ref/fate/jpeg2000dec-ds1_ht_05_b11 |   6 ++
>> tests/ref/fate/jpeg2000dec-ds1_ht_06_b11 |   6 ++
>> tests/ref/fate/jpeg2000dec-hifi_ht1_02   |   6 ++
>> tests/ref/fate/jpeg2000dec-hifi_p1_02    |   6 ++
>> tests/ref/fate/jpeg2000dec-p1_01         |   6 ++
>> tests/ref/fate/jpeg2000dec-p1_02         |   6 ++
>> tests/ref/fate/jpeg2000dec-p1_03         |   6 ++
>> tests/ref/fate/jpeg2000dec-p1_04         |   6 ++
>> tests/ref/fate/jpeg2000dec-p1_05         |   6 ++
>> tests/ref/fate/jpeg2000dec-p1_06         |   6 ++
>> 41 files changed, 361 insertions(+), 1 deletion(-)
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_hm_15_b8
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_02_b11
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_02_b12
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_03_b11
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_03_b14
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_04_b11
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_04_b12
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_05_b11
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_05_b12
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_07_b11
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_07_b15
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_07_b16
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_08_b11
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_08_b15
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_08_b16
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_09_b11
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_10_b11
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_11_b10
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_12_b11
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_14_b11
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_15_b11
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_15_b14
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds0_ht_16_b11
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds1_ht_01_b11
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds1_ht_01_b12
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds1_ht_02_b11
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds1_ht_02_b12
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds1_ht_03_b11
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds1_ht_03_b12
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds1_ht_04_b9
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds1_ht_05_b11
>> create mode 100644 tests/ref/fate/jpeg2000dec-ds1_ht_06_b11
>> create mode 100644 tests/ref/fate/jpeg2000dec-hifi_ht1_02
>> create mode 100644 tests/ref/fate/jpeg2000dec-hifi_p1_02
>> create mode 100644 tests/ref/fate/jpeg2000dec-p1_01
>> create mode 100644 tests/ref/fate/jpeg2000dec-p1_02
>> create mode 100644 tests/ref/fate/jpeg2000dec-p1_03
>> create mode 100644 tests/ref/fate/jpeg2000dec-p1_04
>> create mode 100644 tests/ref/fate/jpeg2000dec-p1_05
>> create mode 100644 tests/ref/fate/jpeg2000dec-p1_06
> 
> on linux x86-32
> 
> --- tests/ref/fate/jpeg2000dec-p1_04 2024-11-10 01:23:28.599497603 +0100
> +++ tests/data/fate/jpeg2000dec-p1_04 2024-11-10 02:38:03.497270241 +0100
> @@ -3,4 +3,4 @@
> #codec_id 0: rawvideo
> #dimensions 0: 1024x1024
> #sar 0: 0/1
> -0,          0,          0,        1,  2097152, 0x9df478aa
> +0,          0,          0,        1,  2097152, 0x5b4dd999
> Test jpeg2000dec-p1_04 failed. Look at tests/data/fate/jpeg2000dec-p1_04.err for details.
> make: *** [tests/Makefile:310: fate-jpeg2000dec-p1_04] Error 1
> make: *** Waiting for unfinished jobs....
> --- tests/ref/fate/jpeg2000dec-ds1_ht_04_b9 2024-11-10 01:23:28.599497603 +0100
> +++ tests/data/fate/jpeg2000dec-ds1_ht_04_b9 2024-11-10 02:38:03.529270529 +0100
> @@ -3,4 +3,4 @@
> #codec_id 0: rawvideo
> #dimensions 0: 1024x1024
> #sar 0: 0/1
> -0,          0,          0,        1,  2097152, 0x9df478aa
> +0,          0,          0,        1,  2097152, 0x5b4dd999
> Test jpeg2000dec-ds1_ht_04_b9 failed. Look at tests/data/fate/jpeg2000dec-ds1_ht_04_b9.err for details.
> make: *** [tests/Makefile:310: fate-jpeg2000dec-ds1_ht_04_b9] Error 1
> 
> thx
> 
> [...]
> -- 
> Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
> 
> Modern terrorism, a quick summary: Need oil, start war with country that
> has oil, kill hundread thousand in war. Let country fall into chaos,
> be surprised about raise of fundamantalists. Drop more bombs, kill more
> people, be surprised about them taking revenge and drop even more bombs
> and strip your own citizens of their rights and freedoms. to be continued
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".

More information about the ffmpeg-devel mailing list