[FFmpeg-devel] [PATCH v2 2/2] ffmpeg: add option -isync

[Gyan Doshi]

On 2022-07-05 09:45 pm, Anton Khirnov wrote:
> Quoting Gyan Doshi (2022-07-04 10:20:22)
>>
>> On 2022-07-04 11:51 am, Anton Khirnov wrote:
>>> Quoting Gyan Doshi (2022-07-02 11:51:53)
>>>> On 2022-07-02 02:12 pm, Anton Khirnov wrote:
>>>>> Quoting Gyan Doshi (2022-07-01 13:03:04)
>>>>>> On 2022-07-01 03:33 pm, Anton Khirnov wrote:
>>>>>>> Quoting Gyan Doshi (2022-06-25 10:29:51)
>>>>>>>> This is a per-file input option that adjusts an input's timestamps
>>>>>>>> with reference to another input, so that emitted packet timestamps
>>>>>>>> account for the difference between the start times of the two inputs.
>>>>>>>>
>>>>>>>> Typical use case is to sync two or more live inputs such as from capture
>>>>>>>> devices. Both the target and reference input source timestamps should be
>>>>>>>> based on the same clock source.
>>>>>>> If both streams are using the same clock, then why is any extra
>>>>>>> synchronization needed?
>>>>>> Because ffmpeg.c normalizes timestamps by default. We can keep
>>>>>> timestamps using -copyts, but these inputs are usually preprocessed
>>>>>> using single-input filters which won't have access to the reference
>>>>>> inputs,
>>>>> No idea what you mean by "reference inputs" here.
>>>> The reference input is the one the target is being synced against. e.g.
>>>> in a karaoke session -  the music track from a DAW would be ref and the
>>>> user's voice via mic is the target.
>>>>
>>>>>> or the merge filters like e.g. amix don't sync by timestamp.
>>>>> amix does seem to look at timestamps.
>>>> amix does not *sync* by timestamp. If one input starts at 4 and the
>>>> other at 7, the 2nd isn't aligned by timestamp.
>>> So maybe it should?
>>>
>>> My concern generally with this patchset is that it seems like you're
>>> changing things where it's easier to do rather than where it's correct.
>> There are many multi=input filters which may be used. amix is just one
>> example.
>>
>> The basic 'deficiency' here is that filters operate upon frames and only
>> look at single frames for the most part, even though frames are part of
>> streams. These streams may have companion streams (which may be part of
>> programs) which are part of a single input. These inputs may have
>> companion inputs.  Anything in this tree may be relevant for a
>> particular operation as a reference, e.g. we have a bespoke filter
>> scale2ref so that we can look at another stream's frames. But we don't
>> have pad2ref, crop2ref ..etc.  So, the absolutely correct thing to do
>> would be to supply a global context to processing modules like
>> filtergraphs , maybe an array of dicts, containing attributes of all
>> inputs like starting time stamps, resolution, string metadata..etc. That
>> would obviate need for these bespoke fields and even filters.
> I don't see how the second paragraph relates to the first one. scale,
> pad, or crop are not multi-input filters, so why are you comparing them

scale is a singe-input filter but scale2ref is a multi-input filter 
which is needed solely because there is no means at present to convey 
info about other streams to a single input filter.
Similarly, we would need a crop2ref, pad2ref..etc to achieve the same 
attribute transfer.  If we had a global context, these counterpart 
filters wouldn't be necessary.


> to amix? I don't think there are so many multi-input filters in lavfi,
> and the issue should be solvable using the same code for all of them.

Because reference about other streams isn't helpful only at the point of 
multi-filter use. One of the streams may want to be resampled to a 
specific rate or sample format based on some user's logic instead of 
letting amix choose one. That's where a global context would help.

>> Actually, this functionality sounds like it sort of existed earlier in
>> the form of map sync (i.e. -map 1:a,0:a:1). Although the assignment
>> syntax still remains (and doesn't warn/error out),  it's a no-op now
>> since the application code was removed in 2012 by Michael, who said he
>> based it off an idea from one of your commits, presumably in Libav.
> So why are you not restoring that functionality and adding a new option
> instead?

I said 'sort of'. That adjustment was implemented in do_video/audio_out, 
so it won't help in filtering, or streamcopying.
This current option adjusts just after demux, so it doesn't have those 
limitations.

Regards,
Gyan