[FFmpeg-devel] [PATCH] fftools/ffmpeg: fix video frame lost sync for the dual input ts files with audio

[lance.lmwang at gmail.com]

From: Limin Wang <lance.lmwang at gmail.com>

How to reproduce the problem(use two ts files with audio stream):
ffmpeg -i left_w_a.ts -i right_w_a.ts -filter_complex "hstack=inputs=2" -f null -

With above command, the audio stream of the second input will be discarded default, however the start_time is 
initalized from the audio stream, so the old code will try to correct the ts offset in the condition which will
cause the video frame out of sync and report some dts error like below:
[null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 43 >= 43
[null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 44 >= 44
[null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 45 >= 45
[null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 46 >= 46
[null @ 0x7fa78c000c00] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 47 >= 47

Or you can try with below command to reproduce the issue:
ffmpeg -i input_w_a.ts -i input_w_a.ts -filter_complex "ssim=f=ssim.log" -f null -

You'll get below ssim result which isn't expected result for the two input is same, the SSIM should be 1.00000
SSIM Y:0.777353 (6.523822) U:0.964087 (14.447539) V:0.966739 (14.780661) All:0.840039 (7.959872)

With the patch applied, the above testing command can get expected result.

other solution are: use the setpts=PTS-STARTPTS filter or skip all audio stream to avoid the timestamp adjustment.

Signed-off-by: Limin Wang <lance.lmwang at gmail.com>
---
 fftools/ffmpeg.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/fftools/ffmpeg.c b/fftools/ffmpeg.c
index 01f04103cf..3fed92639a 100644
--- a/fftools/ffmpeg.c
+++ b/fftools/ffmpeg.c
@@ -4378,13 +4378,16 @@ static int process_input(int file_index)
             && ifile->ts_offset == -is->start_time
             && (is->iformat->flags & AVFMT_TS_DISCONT)) {
             int64_t new_start_time = INT64_MAX;
+            int nb_used_streams = 0;
+
             for (i=0; i<is->nb_streams; i++) {
                 AVStream *st = is->streams[i];
                 if(st->discard == AVDISCARD_ALL || st->start_time == AV_NOPTS_VALUE)
                     continue;
+                nb_used_streams++;
                 new_start_time = FFMIN(new_start_time, av_rescale_q(st->start_time, st->time_base, AV_TIME_BASE_Q));
             }
-            if (new_start_time > is->start_time) {
+            if (nb_used_streams > 1 && new_start_time > is->start_time) {
                 av_log(is, AV_LOG_VERBOSE, "Correcting start time by %"PRId64"\n", new_start_time - is->start_time);
                 ifile->ts_offset = -new_start_time;
             }
-- 
2.21.0