[FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization

[softworkz .]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> Nicolas George
> Sent: Freitag, 11. April 2025 19:09
> To: FFmpeg development discussions and patches <ffmpeg-devel at ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [RFC] FFmpeg Execution Graph Visualization
> 
> Michael Niedermayer (HE12025-03-29):
> > can you repost the AVWriter patches ?
> 
> Sure, but I need to find the latest branch, and it is proving a cause of
> procrastination for me. In the meantime, let me explain anew the
> principles behind it, after a few years I might be able to express them
> better.
> 
> (I hope it means you are considering re-asserting your authority to
> approve it on principle pending review of the code. I will not invest
> more coding into it unless I know nobody has the authority to put it to
> waste with “it does not belong”.)
> 
> 
> 1. Why we need it
> 
> We use a lot of text for small things. As a command-line tool, we use
> text to communicate with users. Even as a library, when GUIs do not have
> the proper widget they will use text, and we need to provide the
> necessary support functions.
> 
> We should have a rule that for every type we add, we must have a
> corresponding to-text API function. Furthermore, these functions should
> all have the same type (up to the type of the opaque pointer), so that
> we can store them in a structure that describes the type along with
> other information (from string, ref/unref, etc.).
> 
> We also need text for inter-process communication, like exporting
> statistics from filters.
> 
> Some of these to-text conversions happen once per frame. That means they
> should happen without dynamic allocation. Quite complex code is
> warranted to prevent dynamic allocations once per frame: see the frame
> and buffer pools too. But some of these to-text conversion have an
> unbounded output. That means the API needs to be able to handle long
> output.
> 
> BPrint meets the criteria: BPrint is as fast as a buffer on the stack
> but can handle strings of arbitrary length.
> 
> Unfortunately, BPrint is ugly. The name is ugly, and more importantly,
> it is inflexible, it works only with flat buffers in memory.
> 
> 
> 2. How AVWriter does it
> 
> AVWriter is an attempt — successful, in my opinion — at keeping what is
> good in BPrint and fixing what is ugly, in particular by giving it
> features similar to AVIO.
> 
> The design principle is the same as AVIO: a structure with callbacks
> that perform the low-level operations.
> 
> The difference with AVIO is that effort were made to keep the structures
> small and allocatable by the caller.
> 
> Also, the back-end is allowed to provide only a few methods: printf-like
> or write-like or get_buffer-like, and the framework will make sure to
> use one that is available. That means quite a bit of code to handle
> doing a printf() on a back-end that only has get_buffer, but it is code
> isolated within a single file and with ample FATE coverage.
> 
> One of the tricks I used to avoid dynamic allocations is to store the
> structure of methods and the structure with an instance as a structure
> with two pointer elements passed by value.
> 
> Another trick I used to avoid dynamic allocations is to store the size
> of a structure in it. That way, if a caller is compiled with an old
> version of the library, it stores a smaller size than the current one,
> and the new version of the library can test and avoid accessing the new
> fields. That lets the caller allocate the structure while keeping the
> ability to add fields to the structure.
> 
> Apart from that, is is rather straightforward. The default AVWriter is a
> simple wrapper around BPrint, but there are also quite a few other
> built-in ones: wrapper around stdio and av_log (without storing the
> whole string), wrapper around a fixed-size buffer. AVwriter can also act
> as a filter: get an AVWriter, you can create a new one that will encode
> to base64 or HTML entities on the fly.
> 
> 
> 3. Where it will go
> 
> The trick passing a struct of methods and the object as a structure with
> two members passed by value is meant to become a very light-weight
> object system. Thus if ctx is a pointer to a structure of type T, we can
> define serialize_t as a structure with functions and consider
> { serialize_t, ctx } as an object of the class “serializable”. But we
> can also make it an object of the class “refcounted” by pairing it with
> another structure.
> 
> The object system, including casts from one class to another, can be
> de-centralized, new classes ca be defined anywhere in the code without a
> central registry. That will be useful to enhance several parts of our
> code:
> 
> - Side data would not need to be added in libavutil. A pair of
>   demuxer-muxer in libavformat can define a new type of side data by
>   defining the methods to handle it.
> 
> - Filters with unusual parsing needs can define a new type of option and
>   plug it into the standard options parsing system. It is extremely
>   useful if the needs of the filter are too specific to add a type in
>   libavutil but too far from something existing to be able to use it
>   conveniently.
> 
> - Pluging new types into the options system will automatically fix our
>   “escaping hell” issue where users sometimes need 42 levels of \ in
>   front of a : to achieve their goals.
> 
> - We can implement a full-featured av_printf() function that serializes
>   on the fly:
> 
>   av_printf(out, "Stream %d at %d Hz, %@ with layout %@\n",
>       st->id, st->sample_rate,
>       avany_sample_fmt(st->sample_fmt),
>       avany_layout(st->channel_layout));
> 
> With this, it becomes possible to implement new features that rely a lot
> on text. That includes:
> 
> - Better error reporting. Instead the error message going to the log,
>   where nobody will see it if it is a GUI, instead of the application
>   having to install a log handler, with the responsibility to assemble
>   lines together and filter out unrelated messages that just happened to
>   arrive at the same time, the error message is stored in the relevant
>   context and can be retrieved cleanly later.
> 
> - Built-in comprehensive documentation. You can do
>   av_get_documentation(ctx, …) and get explanations on how to use it.
>   Depending on the flags given, the returned documentation can be terse
>   and suitable for a tooltip or comprehensive including hyperlinks and
>   reference for the syntax of the options.
> 
> - Serialization of high-level data structures into various standard
>   formats: JSON, XML, etc., precisely what started this thread. That
>   allows to factor the writers in ffprobe and that gives us means to
>   exfiltrate statistics from filters.
> 
> 
> 4. Why it must be in FFmpeg
> 
> I acknowledge that nothing I described is even remotely specific to
> FFmpeg. I could be arguing for AVWriter, a light-weight object system,
> built-in documentation, etc., in a project that does something entirely
> else.
> 
> That means in principle all this could go into a separate library, and
> some people have argued for it. In principle it could, but in practice
> it does not make sense for multiple reasons, but the main reason is
> this:
> 
> As a matter of principle, we do not demand that users install
> third-parties libraries in order to build FFmpeg. We rely on system
> libraries, and we can rely on external libraries for optional features,
> even very important ones (x264…), but if something is needed to run
> FFmpeg at all then it must be in our own code base so that building from
> sources will just work. Strings and error reporting cannot be optional,
> they must be in FFmpeg.
> 
> 
> I will come back later with the code as it was last time.
> 
> Regards,
> 
> --
>   Nicolas George
> _______________________________________________


HI Nicolas,


> - Serialization of high-level data structures into various standard
>   formats: JSON, XML, etc., precisely what started this thread. That
>   allows to factor the writers in ffprobe and that gives us means to
>   exfiltrate statistics from filters.

just FYI - there aren't any writers in FFprobe anymore. Formatting and output are separate things now.

I had asked you about the AVWriter code - even before Michael did - out of interest, for checking out how much the implementation might benefit from AVWriter because I can remind just some fragments about it.
I don't even remember the "object system" aspect - which I find even more interesting, yet the presentation is a bit confusing. I believe that treating them as two separate things would make it a better and easier sell. 

I'm still not sure about the relation between AVWriter and "object-system". Is it that AVWriter is implemented according to that "object-system" proposal, i.e. being like all "objects" are supposed to be implemented in the future?
Or is AVWriter just using the same technique in a similar way but for a different purpose?


I notice how much frustration this subject appears to have caused for you, but you should not project that frustration on the wrong targets.
Like I said before - I don't want to cut you off from the work in the area of text formatting and writing and you are welcome to collaborate - if you want that.
Weirdly enough, I like many of the examples you are giving above. I can say that because I'm not cultivating my own frustrations. If you'd only manage to pretend being a friendly person, that should already suffice to make some good progress. Seriously.

Best,
sw