[FFmpeg-devel] [PATCH] [2/??] [3/3] Filter graphs - Parser for a graph description

[Vitor Sessak]

Hi, and thank everybody for following this!

Aurelien Jacobs wrote:
> Michael Niedermayer wrote:
> 
>> On Tue, Mar 18, 2008 at 11:13:25PM +0100, Aurelien Jacobs wrote:
>>> Michael Niedermayer wrote:
>>>
>>>> On Tue, Mar 18, 2008 at 09:29:51AM +0000, vmrsss wrote:
>>>> [...]
>>>>> (F) If I understood your examples correctly (which I might have
>>>>> not)`, there is still one missing ingredient: a feedback
>>>>> combinator, say !, which takes a filter with at least one input
>>>>> and at least one output stream and feeds back the first output
>>>>> to the first input. For instance, let's look at the examples
>>>>> you have been discussing in this thread:
>>>>>
>>>>> (1)
>>>>>> in --> scale --> crop --> picInPic --> rotate --> split -->
>>>>>> out ^                      |
>>>>>>                             |                      |
>>>>>>                           delay<-------------------/
>>>>>
>>>>> becomes: scale,crop, !( picInPic, rotate, split, delay * nop )
>>>>> -- in fact the duplicated stream from split is fed to delay and
>>>>> delay's output is then fed back to picInPic (if that was the
>>>>> sense of the example, of course...)
>>>> How does the following work in your system?
>>>>
>>>> (in0,tmp0,tmp2)filter1(tmp1,out0,tmp2);(in1,tmp1,tmp3)filter2(tmp3,tmp0,out1)
>>> If I understood correctly, this would look like this:
>>>
>>> !(swap*nop,
>>>      !(swap*nop*nop, nop*swap*nop,
>>>            !(swap*nop,nop*swap,filter1,nop*swap,swap*nop)
>>>          * (swap, !(swap*nop,nop*swap,filter2))
>>>        ),
>>>   swap*nop)
>>>
>>> Well, not very beautiful, but at least, it is possible to describe
>>> such a filter chain.
>> interresting, lets add a 3rd one
>>
>> (in0,tmp0,tmp1,tmp2)filter1(out0,tmp2,tmp8,tmp3);
>> (tmp3,tmp4,in1,tmp5)filter2(tmp7,out1,tmp4,tmp0);
>> (tmp6,in2,tmp7,tmp8)filter3(tmp5,tmp6,tmp1,out2)
> 
> There is no doubt that it's possible to convert it too.
> And there is no doubt that it would be quite messy...
> 
>> And i wonder how many users will be able to do that ...
> 
> Not much, indeed. But I'm not sure many users would really
> understand what this filter setup actually do, even with
> your simple description. I personally have no idea what
> can be achieved with such a filter setup.
> 
>> Also what about filters which can take a variable number of
>> inputs/outputs?
> 
> Do such a beast exist ?

Well, there are only a handful of avfilters that exists now, but I can 
think of something like

(in) split=3 (T1) (T2) (T3);

Also, one can think about a "normalize_timebase" filter that gets n 
streams and make the timestamps of all of them easily described in a 
single timebase.

> Anyway, I guess such a filter would simply accept as many input
> the graph is feed it with.

In a way, this is a problem. Now, the parsing of the filters are 
separated of the initialization (but that may change before hitting 
SVN). But to parse something like

split, nop*rotate, picInPic

There is no way to see the difference between

(in1) ---> split ---> rotate ------+
               |                   picinPic ----> out
               ------> nop    ------+

and

(in1) -----+   +------> rotate
            split
(in2) -----+   +------> nop ------>   picinPic ----> out

without knowing beforehand how many inputs and outputs each filter has.

> 
>>> I suppose this would not exactly be a common filter chain, and I
>>> also suppose that filter1 and filter2 would naturally be designed
>>> so that their "natural" self-feedback in and out pad would be the
>>> first pad. This would highly simplify this filter graph. Here is an
>>> example of the exact same graph, with ideally ordered filters pad:
>>>
>>> (tmp0,tmp1,in0)filter1(tmp0,tmp2,out0);(tmp3,tmp2,in1)filter2(tmp3,tmp1,out1)
>>>
>>> !(swap*nop, !( !filter1 * !filter2, nop*swap*nop, swap,nop,nop))
>>>
>>> So IMO, this case just show that "mathematical" description allows
>>> to represent complex graph as well as the original description.
>> no, it does not, it shows that you could represent the given graph.
>> Its not a proof that all graphs can be represented. Though its quite
>> easy to proof that if you want ...
> 
> Right.
> 
>>> But it don't show which one is the most simple/intuitive to
>>> describe commonly used graph (ie. graph with no feedback or at
>>> most one feedback).
>> Filter chains are just filter1,filter2,filter3 in either system.
>>
>>
>> More complex ones without feedback could look like
>>
>> filter1(T),filter2,filter3,(T)filter4
>> VS.
>> filter1,filter2*nop,filter3*nop,filter4
>>
>> i think first wins
> 
> Maybe, but I don't see much difference.
> 
> Just a very slightly more complex example with no feedback:
> 
> filter1(T1),filter2,filter3,(T2)filter4;(T1)filter5(T2)
> VS.
> filter1,filter2*filter5,filter3*nop,filter4
> 
> To me, second syntax looks slightly more readable.
> 
>> With one feedback
>>
>> (T)filter1,filter2,filter3(T),filter4
>> VS.
>> !(filter1,filter2,filter3),filter4
> 
> both looks equally simple.
> 
>> And with multiple feedbacks we have seen that the !*, description is
>> significantly less readable
> 
> This I agree with.
> But I wonder if multiple feedbacks do have enough practical usage
> that we should care about their readability ?
> 
>> So IMHO while the * operator might be worth considering, the way
>> feedbacks are handled is absolutely messy when there is more than 1
>> feedback.
> 
> Agree.
> BTW, both syntax are not mutually exclusive. Just replace () by []
> in the original syntax and we could combine advantages.
> Just an example with two "branches" plus a feedback:
> 
> (X)filter1(T1),filter2,filter3,(T2)filter4;(T1)filter5(X,T2)
> VS.
> [X]filter1,filter2*filter5[X],filter3*nop,filter4

Looks like a good idea, it also allows dropping the "!" op...

Reading the thread, it looks like that there is no consensus about which 
one is the best. Since mine has already the code written and can be 
later expanded to use the '*' operator, I'll continue the review cycle 
of the code I posted, unless there are any objections.

> 
>> Also i think ,* is unintuitiv |, might be better but then maybe not i
>> dunno
> 
> At first, * seemed not so bad to me, but | might be good too.

Well, I'm have a strong predilection for anything that permit simple 
filters like

rotate=90,scale=320:240

to be passed to ffmpeg without the need of been escaped by the shell. So 
I'm against changing the comma for anything that need escaping.

-Vitor