[FFmpeg-devel] [PATCH 1/2] test: hlsenc: Don't generate test data with -re

[Michael Niedermayer]

On Fri, Aug 07, 2020 at 10:35:58PM +0200, Michael Niedermayer wrote:
> On Thu, Aug 06, 2020 at 10:08:33PM +0300, Martin Storsjö wrote:
> > On Mon, 3 Aug 2020, Martin Storsjö wrote:
> > 
> > > On Mon, 3 Aug 2020, Michael Niedermayer wrote:
> > > 
> > > > is this removing the only test which tests the "-re" flag ?
> > > > if so that would reduce test coverage
> > > 
> > > Well, tehcnically yes, but this test doesn't actually test whether the
> > > "-re" flag does what it's supposed to or anything like that - it only
> > > tests that adding the flag doesn't error out and doesn't alter the
> > > output. 
> 
> yes
> 
> 
> > > And it looks very much unintentional here.
> 
> maybe
> 
> 
> > 
> > Do you want to follow up the discussion here? I'd like at least some sort of
> > acknowledgement back on the discussion before going forward with it in any
> > direction.
> 
> sorry for the lack of reply, theres always more to reply to and do then i seem
> to have time.
> 
> About it not fully testing things, that is of course not great but testing
> things halfway is still better than not testing at all.
> 
> Also it may make sense given the large number of features we have to test
> multiple per test.
> For example with 200 features, it would take 200 individual tests to test each
> in its own test, but only 10 tests if 4 of these would always test the same 
> feature. this of course assumes all kinds of things, its more meant to show
> that a bit of "chaos" in what each test tests could actually improve the
> amount of information we could obtain from the pattern of failing tests

To elaborate here a bit in relation to the context of "-re"

If you have 1 test that tests "-re" in addition to other things like is the
case in master. Its failure will not tell you if its "-re" or something else.
But if 2 tests which are otherwise unlrelated test "-re" in addition then
both failing will give a strong hint its "-re" without further debuging.

So if a test like the -re test is added on top of other tests it should be
to more than 1 if it doesnt have its own specialized test. That way its
easier to identify the cause.
Also obviously for the specific case of "-re" it should be added in such a
way that its not slowing any test down and doesnt cause any time dependant
test failures

So in the end for -re it probably makes sense to have a specialized test
instead as its a bit special and different from other features. But for other
features adding them to some existing tests seems a intriguing idea IMO

thx

[...]

-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

If a bugfix only changes things apparently unrelated to the bug with no
further explanation, that is a good sign that the bugfix is wrong.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20200807/761e4bb1/attachment.sig>