[FFmpeg-devel] [RFC] Scripting filter and foreign language bindings

[Stefano Sabatini]

On date Monday 2013-11-18 17:43:43 +0100, wm4 encoded:
> On Sat, 16 Nov 2013 19:49:33 +0100
> Stefano Sabatini <stefasab at gmail.com> wrote:
[...]
> > Very short followup after some time spent doing some hacks, in case
> > someone is interested.
> > 
> > First, it is clear that in order to be able to do useful stuff in a
> > scripting filter it is required to use the FFmpeg API (e.g. handling
> > frames and stuff), so the first very non trivial problem is to bind
> > the subset of the FFmpeg (public) API to the target scripting
> > language.
> 

> That sounds a bit like overkill. Due to the way libavfilter works,
> maybe it would be best to restrict the beginning to 1.) making it
> possible to use scripting instead of av_expr_* strings to setup
> filters, and 2.) using scripting instead of the lavfi filter language to
> build a filter graph.

1. we don't use expr to setup filters in the first place (also: random
language X should not be employed as substitute for expr)

2. this means binding the lavfi API to a high-level language (I don't
think this was done for any of the available bindings).

> 
> Ideally, there should be some sort of high level API, which should
> basically allow the user to play with frames directly, but this
> wouldn't fit directly into libavfilter's architecture - but a hybrid
> approach might be possible.

But this is not strictly related to libavfilter.
 
> > I did some experiments with SWIG. SWIG can't be considered an
> > universal solution, that is each language will need a specific
> > maintainer and ad-hoc solutions, with or without SWIG, but SWIG can
> > help to simplify the maintainance burden.
> > 
> > See the following discussion for more insight:
> > http://thread.gmane.org/gmane.comp.programming.swig/17211
> 
> Yeah, SWIG sounds like overkill. In any case, if the underlying API is
> somewhat scripting-friendly (memory safety etc.), and if the API small,
> and if it's stable, then creating bindings manually is probably not
> _that_ hard. It might also make sense to pick only one scripting
> language, and stick to it - at least within ffmpeg.

My impression reading the discussion is that SWIG may be the tool, but
it's not an universal tool. It can be used to automate binding
creation, but in general every language has need different needs (for
example SWIG doesn't support Haskell for reasons due to the nature of
the language).

> > One question is if the bindings should be maintained within FFmpeg or
> > if they really should stay outside. I tend to lean towards the second
> > solution. Indeed there are already several external wrappers in
> > various languages, more or less maintained, and it is not realistic to
> > think that there is enough manpower in FFmpeg to keep them udpated,
> > and we should rather leverage the work done by these external
> > contributors.
> 

> I was imagining that scripting support could allow handling new
> use cases, without adding weird things to ffmpeg.c and libav*.

Indeed bindings should be transparent to the usual ffmpeg internal
developer. What I propose is to rely on the work of external bindings,
for example to extend the subset of the supported API. We may start by
adding a section on the wiki, start to use the binding of our
favourite language, and eventually improving the C API to simplify the
binding process. Ideally this should end up improving both bindings
and the library itself.

But I don't think this should be done in the core of the project.
 
> > Another point is that in order to do something useful with scripting
> > we would need to call the internal API, that is what it can't do with
> > an external C filter can't be done with a scripting filter as well, so
> > in order to follow this path we would need to make many internal
> > methods public (and in general make possible for a library user to
> > define a filter and register, register and use it in its own code).
> 
> No, the API should improved. Of course that's hard work and easier said
> than done.
> 

> Custom filters sound good, though. But AFAIK this is against FFmpeg's
> (unwritten?) policy of not allowing external plugins.

I don't think it makes much sense to have a "private" codec or
muxer/demuxer. Filters are different, for custom logic you don't need
much low-level optimizations but rather the availability of external
modules to interface with external libraries and tools (e.g. to query
databases, use widgets and such).
-- 
FFmpeg = Fiendish and Fundamentalist Magic Powerful Evangelical Gnome