[FFmpeg-devel] [RFC] 5 year plan & Inovation

[Andrew Sayers]

On Tue, Apr 23, 2024 at 10:02:58AM +0200, Lynne wrote:
> Apr 23, 2024, 09:47 by ffmpeg-devel at pileofstuff.org:
> 
> > On Tue, Apr 23, 2024 at 02:20:51AM +0200, Michael Niedermayer wrote:
> >
> >> On Thu, Apr 18, 2024 at 10:46:35AM +0200, Stefano Sabatini wrote:
> >> > On date Wednesday 2024-04-17 15:58:32 +0200, Michael Niedermayer wrote:
> >> > > Hi all
> >> > > 
> >> > > The pace of inovation in FFmpeg has been slowing down.
> >> > > Most work is concentarted nowadays on code refactoring, and adding
> >> > > support for new codecs and formats.
> >> > > 
> >> > > Should we
> >> > > * make a list of longer term goals
> >> > > * vote on them
> >> > > * and then together work towards implementing them
> >> > > ?
> >> > > 
> >> > > (The idea here is to increase the success of larger efforts
> >> > >  than adding codecs and refactoring code)
> >> > > It would then also not be possible for individuals to object
> >> > > to a previously agreed goal.
> >> > > And it would add ideas for which we can try to get funding/grants for
> >> > > 
> >> > > (larger scale changes need consensus first that we as a whole want
> >> > >  them before we would be able to ask for funding/grants for them)
> >> > > 
> >> > > Some ideas and why they would help FFmpeg:
> >> > > 
> >> > [...]
> >> > > * client side / in browser support
> >> > >     (expand towards webapps, webpages using ffmpeg client side in the browser)
> >> > >     bring in more users and developers, and it will be costly for us
> >> > >     if we let others take this area as its important and significant
> >> > 
> >> > There are already several projects on github, the most prominent one:
> >> > https://github.com/ffmpegwasm/ffmpeg.wasm/
> >> > 
> >> > In general it would be useful to provide libav* bindings to other
> >> > languages, for example:
> >> > https://github.com/PyAV-Org/PyAV
> >> > https://github.com/zmwangx/rust-ffmpeg
> >> > 
> >> > Not sure these should be really moved to FFmpeg though.
> >>
> >> From a user PoV it would be nice if there was a official
> >> python, rust and wasm binding
> >>
> >> It also would draw in more developers and users to FFmpeg.
> >> test coverage might also improve
> >>
> >> I think the 2 questions are.
> >>  1. is there a binding for some language that wants to become the official
> >>  FFmpeg binding for that language ?
> >>  2. does the FFmpeg community want that too ?
> >>
> >> thx
> >>
> >
> > I've thought about this a lot while trying to learn FFmpeg.
> > IMHO there are two big hurdles to good other-language bindings:
> >
> > First, FFmpeg's interface is full of C idioms that are unintuitive to
> > programmers from other languages.  For example, Stefano Sabatini is
> > patiently explaining to me in anoher thread how contexts are a central
> > concept in FFmpeg's design.  Even where I understood the code on a
> > mechanical level, I had drastically underestimated their importance
> > because I didn't have a mental model to understand them.  Binding
> > FFmpeg functionality in another language is only half the problem -
> > the interface needs to be explained in terms they can understand,
> > or rewritten in terms they already know.
> >
> > Second, the interface is full of special cases that make translation
> > to other languages burdensome.  For example, C errors are based on
> > returning a value and requiring the caller to check it explicitly;
> > whereas most other languages throw an error and allow the caller to
> > catch it or not.  A translator needs to convert every one of those,
> > but FFmpeg functions don't have a standard mechanism to signal the
> > correct behaviour for a given function.  Even the documentation isn't
> > reliably helpful, sometimes saying a variant of "returns an AVERROR",
> > sometimes "returns a negative number", and sometimes it just
> > returns an int and expects the reader to dig through the source.
> > That eats up a huge amount of programmer time, and has to be done for
> > every language that wants a binding.
> >
> > Solving those problems would make it far more practical for translators
> > to make bindings in other languages, and for new people to learn FFmpeg
> > even in C.  For example, creating an `enum AVERROR` and rewriting
> > functions to return it would make the code easier to read and drastically
> > cut translator time.
> >
> 
> We always return a negative number for error. 

This is going to be a lot of detail for a specific example, but I think it
illuminates the general point...

Signalling an error with "a negative number" vs. "an AVERROR" is all-but
synonymous in C - in both cases, you just do `if ( ret < 0 ) return ret;`.  But
the equivalent idiom in most languages involves throwing different data types.
For example, a Python programmer would likely expect the former to throw an
"Exception", but the latter to throw some library-specific "AVErrorException"
type.  A function that "returns a negative number on error" might return `-1`
in all cases, or a random number that resembled an AVERROR in the case you
happened to test, or the former in some versions but the latter in others.
Erring towards throwing `Exception` causes people to avoid the binding because
it obfuscates information they need in their use case, while erring towards
`AVErrorException` causes people to avoid the binding because it generates
misleading error messages.

But let's assume the documentation never says "returns a negative number" when
it means "returns an AVERROR", or says "AVERROR" when it means "...except in
this one specal case where it returns -1".  What does a binding author do with
e.g.  `avformat_network_init()`, which returns an `int` but doesn't say whether
that `int` even indicates an error?  Their choice is to either pick one of the
exceptions or ignore the return code altogether, then wait for angry users to
tell them they got it wrong.  And given that FFmpeg has opted not to guarantee
a specific meaning, any choice will need to be revisited when new versions come
out.

Again, this is just a small example (and I've submitted a patch for the
specific avformat_network_init() issue[0]).  But there are plenty of abandoned
FFmpeg bindings out there, and I suspect it's because of issues like this
generating extra work to write and even more work to maintain.  Solving these
issues in FFmpeg would be at most as much work as solving them in a single
binding, and would also make it easier for new people to learn the C interface.

[0] https://ffmpeg.org/pipermail/ffmpeg-devel/2024-April/325991.html