[FFmpeg-devel] [PATCH] pthread_frame: attempt to get frame to reduce latency

[Derek Buitenhuis]

On 11/03/2020 14:53, Devin Heitmueller wrote:
> Regardless of the actual proposed patch, I think the author's use of
> wallclock time to describe the problem is very reasonable.  I do a
> large amount of work where I'm measuring "glass-to-glass" latency,
> where I am interested in the total pipeline (encode/network/decode),
> and I definitely went through the experience of trying to figure out
> why ffmpeg was introducing nearly 500ms of extra latency during
> decoding.  It turned out that it was because my particular platform
> had 8-cores and thus 16 decoding threads and thus 15x33ms of delay,
> regardless of the stream complexity.

Heavy disagree. The measurement is *specifically* referring to an API call
here, and it *specifically* effects the delay, in frames. The email in question
is conflating timestamps (33ms) per frame with wallclock time later on. It is
not a meaningful comparison to make. Only pain lies down the path of mixing
timestamps and wallclock like that.

Glass-to-glass measurement is important too, but don't conflate the two.

For what it's worth, I pick deterministic delay (in frames! packets-in-to-frames-out)
over the possibility of less delay but non-deterministic every day of the week.
For my own sanity. *Certainly* not as the default and only mode of operation.

> You may not personally care about latency, but there are lots of
> people operating in a world where actual latency matters (e.g. news
> interviews) and they want to be able to use ffmpeg for decoding in
> such environments.  The "problem" is not how many threads are used.
> The problem is "there's way too much latency" and proposed solutions
> include reducing the thread count or changing the heuristic for
> dequeuing frames from the thread pool.

You are again conflating wallclock latency with frame delay latency. Having
a deterministic amount of decoding delay (in frames, packets-in-to-frames-out)
will certainly cause soem wallclock latency, but it *IS NOT* (N x 33ms) (or whatever
timestamp your framerate causes to be the case). It is confusing and not useful to
mix the two concepts like this.

And again, I will choose a deterministic API every day of the week. In this case,
by deterministic I mean 'the exact same code and file/stream will take the same
number of packets in before outputting a frame', not 'it takes the same amount
of time'.

I understand It's important for some usecases to have the fastest glass-to-glass
time, but it's not worth destroying the API with non-determinism to accomplish that.
At the *very least* it should be behind a flag or option so people can choose to
shoot their debugger in the foot by choice.

But mostly I want people to stop conflating timestamps and wallclock, as well as
decoding delay (in frames/packets) vs wallclock-time-toprocess. They're both important
but nobody wins by mixing the two up like this.

- Derek