[FFmpeg-devel] [PATCH] ffmpeg_filter: initialize sub2video.end_pts together with last_pts

[Jan Ekström]

On Fri, Feb 22, 2019 at 3:11 PM Michael Niedermayer
<michael at niedermayer.cc> wrote:
>
> On Thu, Feb 21, 2019 at 01:16:00PM +0200, Jan Ekström wrote:
> > This fixes buffering of samples which causes sudden ballooning of
> > memory usage in case of no subtitle samples coming in for a while if
> > the filter chain had been re-initialized.
> >
> > You can also see messages a la:
> > "Error while add the frame to buffer source(Invalid argument)."
> > disappearing after filter chain re-initializations.
> >
> > Passes fate-sub2video.
> >
> > Example (memory usage before patch around 700+ MiB, after around 150MiB) :
> > /usr/bin/time -v ffmpeg -v verbose \
> >   -i "https://megumin.fushizen.eu/samples/2019-01-18-audio_reconfig_causes_buffer_growth.ts"
> > \
> >   -filter_complex
> > '[0:v:0]yadif=deint=interlaced[yadif_out];[yadif_out][0:s:0]overlay=eof_action=pass:repeatlast=0[overlay_out];[overlay_out]scale=1024:-2[video_out];[0:a:0]aresample=48000:async=1,aformat=channel_layouts=stereo[filtered_audio]'
> > \
> >   -map "[video_out]" \
> >     -c:v mpeg4 \
> >     -b:v 750k \
> >   -map "[filtered_audio]" \
> >     -c:a aac \
> >     -b:a 192k \
> >   "test.mp4"
> >
> > Best regards,
> > Jan
>
> >  ffmpeg_filter.c |    1 +
> >  1 file changed, 1 insertion(+)
> > 5f88558fef759023173b4c8efe157aa30fc9e337  0001-ffmpeg_filter-initialize-sub2video.end_pts-together-.patch
> > From 9c824c36c972aca19f2747437c8edc71b6c0886c Mon Sep 17 00:00:00 2001
> > From: =?UTF-8?q?Jan=20Ekstr=C3=B6m?= <jeebjp at gmail.com>
> > Date: Thu, 20 Feb 2019 20:54:11 +0200
> > Subject: [PATCH] ffmpeg_filter: initialize sub2video.end_pts together with
> >  last_pts
>
> Breaks: (video stream is empty after this) when a duration is used
> ./ffmpeg -i in.mkv  -filter_complex '[0:s:1]scale=800:600' -t 15 -qscale 2 test.avi
>
> both video and subtitles appear in the first 5 seconds
>
> input is a pgs in mkv file.
> If you cannot reproduce with a random file then say so and ill try to
> turn the file i have into a small testcase i can share
>
> thanks
>

I can replicate this.

Old behavior is kept if end_pts is initialized to its previous
implicit initialization value of zero.

I initialized this originally as INT64_MIN because that way it was
matching, and there were some comments about negative PTS values in
the commit that added the other value's initialization.

In my testing earlier I did find both values working for the use cases
I tested with, so I will send out a patch to switch around to zero
initialization for now, as the main part for not causing possibly
infinite buffering in lavfi is that when sub2video_heartbeat gets
called after a filter chain (re-)initialization sub2video.end_pts is
less than INT64_MAX.

This might be an issue with something expecting timestamps that start
from zero for some sort of "vsync", though, as the heartbeats are
nicely sent out and we are getting actual AVFrames pushed into the
filter chain. The major difference seems to be that after X frames
failed requests start appearing in the filter chain, and the current
sub2video AVFrame gets fed into the filter chain again. Additionally,
the ffmpeg.c vsync logic seems to not duplicate or drop frames when
starting from INT64_MIN.

My debugging av_log code for sub2video can be found at
https://github.com/jeeb/ffmpeg/commit/f1750f489be345f74525f7a2ad89e40c4d7e0493
.

Jan

Excerpt by git diff --no-index --word-diff:
diff --git a/pushing_log_before.log b/pushing_log_int64_min.log
index c163c36c37..c521380b37 100644
--- a/pushing_log_before.log
+++ b/pushing_log_int64_min.log
@@ -1,86 +1,74 @@
[mpegts @ [-0x2fe5b00]-]{+0x263eb00]+} sub2video: using 1920x1080 canvas
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS -1
(last_pts: -9223372036854775808, end_pts:
[-0).-]{+-9223372036854775808).+}
sub2video: Update for stream 2, (pts: [-0,-]{+-9223372036854775808,+}
end_pts: 9223372036854775807, num_rects: 0)
sub2video: Pushing a 1920x1080 frame to 'graph 0 input from stream
0:2' with PTS [-0-]{+-9223372036854775808+}
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS
449 (last_pts: [-0,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS
899 (last_pts: [-0,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS
1349 (last_pts: [-0,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS
1799 (last_pts: [-0,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS
2249 (last_pts: [-0,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS
2699 (last_pts: [-0,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS
3149 (last_pts: [-0,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS
3599 (last_pts: [-0,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS
4049 (last_pts: [-0,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS
4499 (last_pts: [-0,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS
4949 (last_pts: [-0,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS
5399 (last_pts: [-0,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS
5849 (last_pts: [-0,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).
sub2video: Heartbeat[-found 1 failed requests. Feeding filter chain again-]
[-sub2video: Pushing a 1920x1080 frame to 'graph 0 input from stream
0:2' with PTS 5849-]
[-sub2video: Heartbeat-] from stream 1 (audio, PID: 4352) stream at
PTS 6299 (last_pts: [-5849,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS
6749 (last_pts: [-5849,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS
7199 (last_pts: [-5849,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS
7649 (last_pts: [-5849,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).
sub2video: Heartbeat from stream 1 (audio, PID: 4352) stream at PTS
8099 (last_pts: [-5849,-]{+-9223372036854775808,+} end_pts:
9223372036854775807).