[FFmpeg-devel] [RFC PATCH 0/2] Make fate tests succeed with zlib-ng

[Anton Khirnov]

Quoting Derek Buitenhuis (2024-10-18 16:55:58)
> On 10/18/2024 1:02 PM, Alexander Strasser via ffmpeg-devel wrote:
> > Alexander Strasser (2):
> >   Reapply "tests/fate: disable compression for zlib-based codecs"
> >   fate: Make it possible to have alternative reference files
> 
> I want to add this here, since I brought it up on IRC.
> 
> I think this entire saga has been pretty silly, and a workaround for
> what is fundementally a bad test. Testing the exact contents and size
> of encoded output here is fundementally the wrong thing to test and
> provides no value as a test. I'd argue it provides negative value.
> 
> For a lossles encoder, especially one that uses external deps, but this
> applies to internal ones too, almost no value is provided by ensuring
> that its output is bit-exact - I'd actually consider it negative value.
> It's testing the wrong thing (that output remains correct, decodable,
> metadata, etc.) and only ensures no changes to anything can ever be
> made without a reference update.

+1

-- 
Anton Khirnov