[FFmpeg-devel] [PATCH v2 2/2] ffmpeg: add option -isync

[Gyan Doshi]

On 2022-07-05 10:54 pm, Anton Khirnov wrote:
> Quoting Gyan Doshi (2022-07-05 19:10:33)
>>
>> On 2022-07-05 09:45 pm, Anton Khirnov wrote:
>>> Quoting Gyan Doshi (2022-07-04 10:20:22)
>>>> On 2022-07-04 11:51 am, Anton Khirnov wrote:
>>>>> Quoting Gyan Doshi (2022-07-02 11:51:53)
>>>>>> On 2022-07-02 02:12 pm, Anton Khirnov wrote:
>>>>>>> Quoting Gyan Doshi (2022-07-01 13:03:04)
>>>>>>>> On 2022-07-01 03:33 pm, Anton Khirnov wrote:
>>>>>>>>> Quoting Gyan Doshi (2022-06-25 10:29:51)
>>>>>>>>>> This is a per-file input option that adjusts an input's timestamps
>>>>>>>>>> with reference to another input, so that emitted packet timestamps
>>>>>>>>>> account for the difference between the start times of the two inputs.
>>>>>>>>>>
>>>>>>>>>> Typical use case is to sync two or more live inputs such as from capture
>>>>>>>>>> devices. Both the target and reference input source timestamps should be
>>>>>>>>>> based on the same clock source.
>>>>>>>>> If both streams are using the same clock, then why is any extra
>>>>>>>>> synchronization needed?
>>>>>>>> Because ffmpeg.c normalizes timestamps by default. We can keep
>>>>>>>> timestamps using -copyts, but these inputs are usually preprocessed
>>>>>>>> using single-input filters which won't have access to the reference
>>>>>>>> inputs,
>>>>>>> No idea what you mean by "reference inputs" here.
>>>>>> The reference input is the one the target is being synced against. e.g.
>>>>>> in a karaoke session -  the music track from a DAW would be ref and the
>>>>>> user's voice via mic is the target.
>>>>>>
>>>>>>>> or the merge filters like e.g. amix don't sync by timestamp.
>>>>>>> amix does seem to look at timestamps.
>>>>>> amix does not *sync* by timestamp. If one input starts at 4 and the
>>>>>> other at 7, the 2nd isn't aligned by timestamp.
>>>>> So maybe it should?
>>>>>
>>>>> My concern generally with this patchset is that it seems like you're
>>>>> changing things where it's easier to do rather than where it's correct.
>>>> There are many multi=input filters which may be used. amix is just one
>>>> example.
>>>>
>>>> The basic 'deficiency' here is that filters operate upon frames and only
>>>> look at single frames for the most part, even though frames are part of
>>>> streams. These streams may have companion streams (which may be part of
>>>> programs) which are part of a single input. These inputs may have
>>>> companion inputs.  Anything in this tree may be relevant for a
>>>> particular operation as a reference, e.g. we have a bespoke filter
>>>> scale2ref so that we can look at another stream's frames. But we don't
>>>> have pad2ref, crop2ref ..etc.  So, the absolutely correct thing to do
>>>> would be to supply a global context to processing modules like
>>>> filtergraphs , maybe an array of dicts, containing attributes of all
>>>> inputs like starting time stamps, resolution, string metadata..etc. That
>>>> would obviate need for these bespoke fields and even filters.
>>> I don't see how the second paragraph relates to the first one. scale,
>>> pad, or crop are not multi-input filters, so why are you comparing them
>> scale is a singe-input filter but scale2ref is a multi-input filter
>> which is needed solely because there is no means at present to convey
>> info about other streams to a single input filter.
>> Similarly, we would need a crop2ref, pad2ref..etc to achieve the same
>> attribute transfer.  If we had a global context, these counterpart
>> filters wouldn't be necessary.
> In my experience, global *anything* is almost always a sign of bad
> design and only leads to pain and suffering. The proper solution in this
> case would be making the filtergraph construction API more flexible.
> Then the code that actually has all the necessary information (i.e.
> ffmpeg.c or other library caller) would set the filter parameters
> however you want. Then none of these whatever2ref hacks would be needed.

Some of the context data will be used by filters during runtime. So, a 
flexible API could help during init but not afterwards. The context has 
to be accessible during lifetime of filters.

About this patch, the user can already add a custom ts offset to an 
input but it has to be a pre-specified fixed constant. This patch allows 
the user to set one relative to another input. That can only be done in 
ffmpeg.c after all inputs have been opened.

Regards,
Gyan