[FFmpeg-devel] [RFC] AVFilter Parser

[Vitor Sessak]

Hi

vmrsss wrote:
> Sorry guys, for some reason my email is badly asynchronous: I receive  
> your emails with large delays, which makes the discussion a bit  
> clumsy. In particular,  when I wrote my previous proposal I had not  
> seen several emails that apparently have been sent before it. This is  
> somehow embarrassing.
> 
> Below, I'll comment on the new (for me) messages
> 
>>> How do
>>> you express that out_k is to be fed back to in_h? Or can you only do
>>> feedback on intermediate labels?
>> (out_k) is one of the outputs of the whole filter graph. It can't be
>> feed to anything. Maybe I don't understand your question.
> 
> The idea to keep feedback labels entirely separated from input and  
> output labels is entirely right. But should I not bee able to take a  
> filter and perform some feedback after composing as part of a larger  
> graph? I'll say more below.
> 
>>> Which one? the first? or the last? the one missing from the list?
>>> Perhaps I should know the answer, but I am not clear about your
>>> intended use of labels. Let me take an example you use below:
>>>
>>> 	 '(in1)(in2) picInPic, (in3) picInPic(out1)'
>>>
>>> Is "(in1)(in2)" the same as "(in2)(in1)" ? I expect the answer is no,
>>> the second form would swap the two input stream. And (in3) refers  
>>> to a
>>> third input from the entire chain. Is that fed to the second input of
>>> picInPic?
>> You got all that right.
>>
>> In order to feed it to the first input, would I have to
>>> write the filter this way:
>>>
>>> 	 '(in1)(in2) picInPic(tmp), (in3)(tmp) picInPic(out1)'  ?
>> Unless this is a typo, you got this one wrong. The comma would link  
>> the
>> second (inexistent!) output of picInPic to the first input of the  
>> other
>> picInPic. It would cause a syntax error. Maybe you meant
>>
>> '(in1)(in2) picInPic(tmp); (in3)(tmp) picInPic(out1)'  ?
> 
> Well, I simply meant to ask: the (1st) output from the 1st picInPic is  
> fed to the 1st input of the 2nd picInPic. How do I specify that I want  
> that output to go to the 1st input, and (in3) to the 2nd?

No. The first and only output of the first picInPic is fed to the second 
input of the second picInPic. If you wanted the other way around, you 
could do

'(in1)(in2) picInPic(tmp); (tmp)(in3) picInPic(out1)'

or more simply

'(in1)(in2) picInPic, (in3) picInPic(out1)'

> 
> 
>> mygraph := (in1) vflip (out1)
>>
>> vitor at vitor$ ffmpeg -i file1.avi -vfilters '(in1) mygraph (out1)'  
>> file2.avi
>>
>> And yes, textual substitution will give
>>
>> '(in1) (in1) vflip (out1) (out1)' which is nonsense.
> 
> Ah! This I don't understand:
> 
> First of all, in -vfilters '.....' (in1) and (out1) carry absolutely  
> no information, why would to use them? Only one stream, no confusion  
> is possible: just write -vfilters 'mygraph'.

I want to support that, but to clarify the discussion I always put in my 
examples the "(in#)" token, even when they are optional.

> 
> Then, do you or do you not allow (x) mygraph (y) ? 

Not as

vitor at vitor$ ffmpeg -i file1.avi -vfilters '(x) mygraph (y)'  file2.avi

This is a syntax error. If you are wondering where the "(in)" token may 
be useful, there are things like

vitor at vitor$ ffmpeg -i file1.avi -vfilters 'vsrc_movie=movie.avi, crop, 
pad, vflip, (in1) picInPic (out1)'  file2.avi

In this case I'm against making it optional.

> I have the feeling  
> you are arguing against the substitution, not again that form. (But  
> then, if (in1) mygraph (out1) is non-sense after the substitution,  
> isn't it non-sense also before?)

No. "(in1) (in1)" is always nonsense. "(in1)" can appears just once per 
context.

> Of course there is an important reason to allow (x) mygraph (y): it is  
> that you want to capture (that is name) its input or output to do some  
> non-trivial re-routing of the streams: eg, I might want to use  
> mygraph, feed it with the second of my input streams, and then send  
> its output as 2nd input of another filter. How do you propose I do  
> that? I think it would be natural to write something of the kind:
> 
> 	(in2) mygraph (a) ; (in1)(a) picInPic (out1)

This is well formed in my grammar.

> Why not? Do you agree? Also, getting back the point above about  
> feedback, what would be wrong with something like:
> 
> 	(a)(in1)picInPic , mygraph (a)
> 
> Why not? Is that  allowed?

That's well formed also, but has no outputs, unless now mygraph has two 
output streams.

>>> with respect to the form above (A)(B)(C) my_filter1 (out1), if I make
>>> a textual substitution I find myself with
>> As I've already said, textual substitution is not needed (and not
>> wanted, at least by me). It's not easier to parse and it's not  
>> easier to
>> understand. So, unless you give a good argument for textual
>> substitution, please stop using it in examples or arguments for how  
>> the
>> syntax should be.
> 
> I don't think I am making an argument for textual substitution, I am  
> trying to discuss mechanisms for input/output binding. I am not  
> thinking about implementation, I am thinking about combinators, and I  
> would like to be able to understand them also at the level of  
> syntactic manipulation.
> 
>>> Because of course, you'll have situations like this:
> 
>>> 	my_f1 = '(in1)something(out1)'
>>> 	my_f2 = '(in2)otherthing(out1)'
>>>
>>> 	movie=f1.avi(A);
>>> 	movie=f2.avi(B);
>>> 	(A)(B) my_f1 ; my_f2 (out1)(out2)
>>>
>> no. Syntax error. You defined
>>
>>         my_f1 = '(in1)something(out1)'
>>
>> That means you defined a filter that gets one (and only one) input,
>> filter it with the filter "something" and gives one (and only one)
>> output. In
>>
>>        (A)(B) my_f1 ; my_f2 (out1)(out2)
>>
>> you are passing two outputs to my_f1. Maybe you meant
>>
>>  	my_f1 = '(in1)something(out1)'
>>  	my_f2 = '(in2)otherthing(out1)'
>>
>>  	movie=f1.avi(A);
>>  	movie=f2.avi(B);
>>  	(A) my_f1 (out1); (B) my_f2 (out2)
> 
> I'd expect "(A)(B) (my_f1 ; my_f2) (out1)(out2)" and "(A) my_f1  
> (out1); (B) my_f2 (out2)" to be exactly the same (perhaps I should use  
> * here instead of ; ?) And here is why: "my_f1;my_f2" has got two  
> input and two streams A (out1) is the first input (output), B (out2)  
> the second input (output). The same happens to (A) my_f1 (out1); (B)  
> my_f2 (out2)

I think here you misunderstood the original meaning of my semi-colon. 
More on this later.

> 
> (There is however something else I don't understand with (B) my_f2  
> (out2): can you use out2? Or should it be out1? There is only one  
> output there.)

Not always.

vitor at vitor$ ffmpeg -vfilters 'movie=f1.avi(A); movie=f2.avi(B); (A) 
my_f1 (out1); (B) my_f2 (out2)'  file1.avi file2.avi

the output of my_f1 will go to file1.avi and that of my_f2 go to 
file2.avi. It is the same as

vitor at vitor$ ffmpeg -vfilters 'movie=f1.avi(A); movie=f2.avi(B); (A) 
my_f1 (out2); (B) my_f2 (out1)'  file2.avi file1.avi

>> Also, looking at
>>
>>        (A)(B) my_f1 ; my_f2 (out1)(out2)
>>
>> It gives me the impression you misunderstood the meaning in my  
>> syntax of
>> the semi-colon.
> 
> Yes, I am clearly confused about your syntax. Sorry. I am trying to  
> understand.
> 
>> For such a line be valid, my_f1 can't have any output.
>> I'll try to explain by a drawing:
>>
>> (etc) filter (etc2) ; (tmp1) filter2 (tmp2)
>>
>> means
>>
>> (etc) ------> filter ---> (etc2)       (tmp1) --->  filter2 ---->  
>> (tmp2)
>>
>> another example
>>
>> (etc) dead_end_filter ; vsrc_movie=a.avi (tmp2)
>>
>> (etc) ----> dead_end_filter     vsrc_movie ---> (tmp2)
>>
>> (mind the difference with the coma)
>>
>> The semi-colon describes two different parts of the filter chain  
>> without
>> doing any linking. As I said,
>>
>>        vflip ; hflip
>>
>> is a syntax error no matter where it appears. It expects vflip to have
>> no output, but it has one. (Yes, I know it is related with your '*',  
>> but
>> it has the lowest precedence)
> 
> This I don't understand: everybody knows what these filters are like,  
> that is a perfectly fine filter with two inputs (first to vflip,  
> second to hflip) and two outputs (first from vflip, second from  
> hflip). Why do you want me to over-specify (in1)vflip(out1); 
> (in2)hflip(out2) ? As I said above, the only reason I might want to do  
> that is if I need to name the output for some non-trivial threading.

As I said in the beginning of the message, that was the first syntax I 
tried to describe to the mailing list. I agree it's better if I add a 
'*' operator. I proposed it as extensions (1) and (2), based on your syntax.

>>> I think it is important that whatever the syntax is, users are  
>>> allowed
>>> to omit elements so as to write simple filters as simply as always,  
>>> eg:
>>>
>>> 	... remove_logo,deinterlace,crop,scale,pad...
>>>
>>> In this sense, it's important to be able to leave streams implicit if
>>> possible.
>> Yes, I'm favorable to that. But how is this related to this  
>> suggestion?
> 
> Only, is this allowed by your syntax?

Well, if the first filter described is missing k inputs, auto-insert 
"(in1) ... (in_k)". Same for outputs in the end.

>> Which convention for implicit linking?
>>
>> The comma links the last stream of the previous filter to the first of
>> the next. This is explicit and well defined.
> 
> Well, implicit means that it is not explicitly noted. I don't  
> particularly like it, but after all I don't mind. What I still don't  
> understand is what do I need to do in order to send the last stream of  
> the previous filter somewhere else. Is the answer: use ";" ? That'd be  
> quite an overhead, 

Yes, it is quite an overhead. That why I though of extension (3), based 
on Michael's syntax.

-Vitor