[FFmpeg-devel] [RFC] Switching ffmpeg.c to a threaded architecture

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> Anton Khirnov
> Sent: Tuesday, April 5, 2022 9:46 PM
> To: FFmpeg development discussions and patches <ffmpeg-
> devel at ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [RFC] Switching ffmpeg.c to a threaded
> architecture
> 
> Quoting Michael Niedermayer (2022-04-05 21:15:42)
> > On Mon, Apr 04, 2022 at 01:29:48PM +0200, Anton Khirnov wrote:
> > > Hi,
> > > this WIP patchset is the first major part of my ongoing work to
> change
> > > ffmpeg.c architecture such that every
> > > - demuxer
> > > - decoder
> > > - filtergraph
> > > - encoder
> > > - muxer
> > > lives in its own thread. The advantages of doing this, beyond
> increased
> > > throughput, would be enforced separation between these components,
> > > making the code more local and easier to reason about.
> > >
> > > This set implements threading for muxers. My tentative plan is to
> > > continue with encoders and then filters. The patches still need
> some
> > > polishing, especially the last one. Two FATE tests do not yet
> pass, this
> > > will be fixed in later iterations.
> > >
> > > Meanwhile, comments on the overall approach are especially
> welcome.
> >
> > I agree that cleanup/modularization to make the code easier to
> > understand is a good idea!
> > Didnt really look at the patchset yet.
> > I assume these changes have no real disadvantage ?
> 
> Playing the devil's advocate, I can think of the following:
> 1) ffmpeg.c will hard-depend on threads
> 2) execution flow will become non-deterministic
> 3) overall resource usage will likely go up due to inter-thread
>    synchronization and overhead related to new objects
> 4) large-scale code changes always carry a higher risk of regressions
> 
> re 1): should not be a problem for any serious system
> re 2): I spent a lot of effort to ensure the _output_ remains
>        deterministic (it actually becomes more predictable for some
>        cases)
> re 3): I expect the impact to be small and negligible, respectively,
> but
>        would have to be measured once the conversion is complete
> re 4): the only way to avoid this completely would be to stop
>        development
> 
> Overall, I believe the advantages far outweigh the potential
> negatives.

Hi,

do I understand it right that there won't be a single-thread
operation mode that replicates/corresponds the current behavior?

Not that I wouldn't welcome the performance improvements, but one
concern I have is debugging filtergraph operations. This is already 
a pretty tedious task in itself, because many relevant decisions 
are made in sub-sub-sub-sub-sub-functions, spread over many places.
When adding an additional - not even deterministic - part to the 
game, it won't make things easier. It could even create situations
where it could no longer be possible to replicate an error in a 
debugger - in case the existence of a debugger would cause a variance
within the constraints of the non-determinism range. 

From another point of view, this is a change, so fundamental like
ffmpeg(.c) hasn't seen in a long time.
I would at least suppose that this could cause issues at many ends,
and from experience, there may be additional ends where it's rather
unexpected to  have effects.

In that context, I think that doing a change of such a wide scope
in an irreversible way like this, would impose quite a burden on
many other developers, because sooner or later, other developers
will run into situations where something is no longer working like 
before and you'll regularly wonder whether this might be a consequence
of ffmpeg.c threading change or caused by other changes.
But then, you won't be able anymore to bisect on that suspicion,
because the threading change can't be reverted and (as long as it's
not shortly after the change) there might have been too many other 
changes to easily port them back to a state before the threading
change.

I wonder whether this couldn't be done in a way that the current
behavior can be preserved and activated by option?

Wouldn't it be possible to follow an approach like this:

- Assuming the code would be fine and it would mark the desired 
  end result
- Put it aside and start over from the current HEAD
- Iteratively morph the code current code in a (possibly) long
  sequence of refactoring operations where every single one
  (and hence in sum) are semantically neutral - until the code
  is turned more and more into what has already been developed
- eventually, only few differences will be left, and these can 
  be made switchable by an option - as a result, both - old and
  new operation modes would be available.

I don't know whether there's a name to this approach, probably 
there is, yet I never cared. Way more important is that I always 
had good results following this methodology.
The funny thing about it is, that when you have a reliable tooling
for refactoring, you can even stop thinking (well, sort of..)
while transforming the code. Also, when you can't imagine how
the end result would look like or wonder whether it would 
work out at all, it's fun to watch the morphing (if you're 
doing it no-brain-wise)

softworkz