[FFmpeg-devel] RFC: opinions on making fftools work like a library

[Stefano Sabatini]

On date Sunday 2023-03-19 11:01:55 +0200, Rémi Denis-Courmont wrote:
> Le lauantaina 18. maaliskuuta 2023, 13.33.36 EET Neal Gompa a écrit :
> > > Maybe you should first clarify what exactly do you mean by "work like a
> > > library". Meaning literally turn it into a library with a C API you can
> > > call? That would be a massive amount of work, though my current
> > > refactorings are moving in that direction.
> > > Or provide stronger guarantees for commandline options? Or something
> > > else?
> > 
> > I think the idea is to turn fftools (e.g. ffmpeg/ffplay/ffprobe) into
> > a generic, simple high level stable API that consumers can use. The
> > raw libav* APIs are difficult for most people to use, which leads to a
> > ton of wrappers around the fftools instead.
> 
> That sounds like a self-contradictory goal, TBH.
> 
> There are so many different settings, complex interactions between settings, 
> and so much variability depending on versions and build parameters.
> 
> > Yes, it would be a lot of work, but it would make using ffmpeg way easier.
> 
> Or not. You would have to define an API for literally every command line switch 
> that each of the tools exhibits. This would unavoidably result in some kind of 
> high-level abstract and rather complex settings API that would be prone to 
> silent breaks has the names, values and semantics of settings vary with 
> versions and builds.
> 
> There may be value in doing that anyway. In particular, it could improve the 
> progress and error reporting with callbacks rather than parsing the standard 
> output. But it probably wouldn't be in "simplifying" the use of the tools.

One of the ideas at some point was to move the functionality into kind
of high-level constructs such as a libavfilter filtergraph: you would
have basically to define the input and output of the filtergraph and
then feed and read the output from it.

At the moment this has several limitations though: for example there
is all kind of custom logic into ffmpeg.c (e.g. related to timestamp
handling and global synchronization). This in theory could be achieved
by extending the filtering logic to data, so that you have pluggable
filters to handle custom data processing logic (e.g. timestamp
tweaking).

Then you could implement a player by reading from the filtergraph and
sending the output packets to your rendering engine and sending
controls to the filtergraph to stop/resume processing and changing the
filters parameters. ffprobe could be implemented as a data
filter/processor on top of a decoding filtergraph.

This is an ideal design but in practice there are still many things
missing (for example data filtering) and threading/synchronization
with the application logic might be tricky.