[FFmpeg-devel] [PATCH 0/9] A New Take on the Immortal Sub2Video Code
Soft Works
softworkz at hotmail.com
Thu Aug 19 10:43:06 EEST 2021
Hi,
some of you might remember that I had warmed-up the discusion about
improving support for subtitle handling in ffmpeg sometime last year.
A lot of different voices were heard and the unfinished work from
Matthieau Buron has been in focus. My takeaway was that it's a pretty
difficult subject involving complex requirements, for which there is no
simple solution in sight. Consequently, I had put this on hold at my side
just like all others.
Few days ago I had to revisit and re-evaluate the subtitle situation once
again: the current sub2video "hack" implementation is slow and
ineffective: it creates a full-size video frame from the source subtitles
and that full-size frame needs to be overlaid/blended with/onto every
single video frame. That's a pretty expensive operation and it needs to be
performed for every single video frame, even when there's nothing to
overlay at all (= no subs to show in current scene).
And when there are subtitles to show, those are usually limited to smaller
areas on the screen; again, there's no need to overlay a full-size image
onto each frame. From a performance perspective, it would be much more
efficient to overlay the individual rects onto each frame only instead of
a full frame blend.
From a general perspective - even though it has always been considered as
a 'hack' - the sub2video implementation has become a kind of standard
behavior, despite of its shortcomings. Users have accommodated with the
respective command lines and are expecting these to work. I'd expect any
deviation from the current behavior to be filed as regression bugs.
From there, I had decided to go for a small-step improvement of that
situation while staying inside these corner points, and the plan was simple:
- keep the current sub2video functionality alive at its core
- but stop creating full video frames (with alpha) for feeding into a
filtergraph (and later overlay)
- instead, use refcounting for AVSubtitle and attach it to frames of (new)
type AVMediaType_Subtitle
- Those subtitle frames can travel through a filtergraph without needing
to materialize them to images first
For more efficient overlay (processing the relevant rects only), I have
created a new filter: overlay_subs:
- Input0: video
- Input1: subtitles (format: SUBTITLE_BITMAP)
- Output0: video
In order to keep compatibility with existing command lines, I have added
another filter: sub2video (reminescent naming)
sub2video creates video frames (with alpha) from an existing subtitle
stream:
- Input0: subrtitles (format: SUBTITLE_BITMAP)
- Output0: video
This filter gets auto-inserted to retain compatibility with current
sub2video command lines.
As AVSubtitle can carry both, textual and graphical subtitles, the ability
for sending textual subtitles through a filtergraph came more or less for
free.
For testing purposes, I have added another filter: sleet (translates
subtitles to 'L337 speak')
- Input0: subtitles (format: SUBTITLE_ASS or SUBTITLE_TEXT)
- Output0: subtitles (format: same as input)
Why leet? Real world use is surely questionable, but it has two advantages
that make it a great choice for testing:
You can see from almost every single line whether it has been filtered,
and the text is still readable which allows to verify that timings are
still valid.
Working Command Lines
Using "overlay_subs" (better performance):
-y -loglevel verbose -ss 00:02:30 -i INPUT -filter_complex
"[0:0]format=bgra[main];[main][0:3]overlay_subs=x=0:y=-220:format=auto,forma
t=yuv420p" output.mp4
Using regular Overlay and the new sub2video filter:
-y -loglevel verbose -ss 00:02:30 -filter_threads 1 -i INPUT
-filter_complex
"[0:0]format=bgra[main];[0:3]sub2video[subs];[main][subs]overlay=x=0:y=-220:
format=auto,format=yuv420p" output.mp4
Legacy command line compatibility:
(using regular overlay; sub2video filter is inserted automatically)
-y -loglevel verbose -ss 00:04:30 -filter_threads 1 -i
INPUT -filter_complex
"[0:0][0:3]overlay=x=0:y=-220:format=auto,format=yuv420p" output.mp4
To make this work, I roughly did the following:
- Added a new property 'type' to AVFrame of type enum AVMediaType
- Changed code that uses width/height for checking whether a frame is
video or audio and check the 'type' property instead
- Added ff_ssrc_sbuffer and ff_ssink_sbuffer filters
- Changed filtergraph code to support frames with AVMediaType ==
AVMediaType_Subtitle
- Transformed the existing sub2video code accordingly, while still keeping
the heartbeat mechanism in place
Next Steps
- Create modified version of the ASS filter that works based on (text)
subtitle filter input
- Create modified EIA-608/708 filter (e.g. split_eia608) which has one
video input and one video output, plus a second output of type 'subtitle'
(format: text or ass)
This will allow to burn-in eia subs or save them in any other subtitle
format without needing the weird "movie:xxx" input filter construct.
- First of all: Get this patchset right...
Reaching out for Help...
This is a huge task where I hadn't expected to get that far. I'd consider
my ffmpeg expertise as "spot knowledge": I do know certain areas very
well, but there are others that I'm less familiar with.
Nonetheless, I have a strategy and a working proof of concept.
I'm, sure that there are quite a few things that need adjustment, and I'd
be glad about any help and assistance!
Thanks,
softworkz
---
softworkz (9):
lavu/frame: avframe add type property
avfilter/subtitles: Add subtitles.c
avfilter/avfilter: Handle subtitle frames
avfilter/overlay_subs: Add overlay_subs filter
avfilter/sub2video: Add sub2video filter
avfilter/sbuffer: Add sbuffersrv and sbuffersink filters
avfilter/sleet: Add sleet filter
fftools/ffmpeg: Replace sub2video with subtitle frame filtering
Adjust sub2video Fate tests
configure | 2 +-
fftools/ffmpeg.c | 324 +++---
fftools/ffmpeg.h | 7 +-
fftools/ffmpeg_filter.c | 198 +++-
fftools/ffmpeg_hw.c | 2 +-
fftools/ffmpeg_opt.c | 3 +-
libavfilter/Makefile | 6 +
libavfilter/allfilters.c | 13 +-
libavfilter/avfilter.c | 8 +-
libavfilter/avfiltergraph.c | 5 +
libavfilter/buffersink.c | 65 ++
libavfilter/buffersink.h | 15 +
libavfilter/buffersrc.c | 65 ++
libavfilter/buffersrc.h | 1 +
libavfilter/formats.c | 11 +
libavfilter/internal.h | 8 +
libavfilter/sf_sleet.c | 209 ++++
libavfilter/subtitles.c | 61 ++
libavfilter/subtitles.h | 42 +
libavfilter/svf_sub2video.c | 260 +++++
libavfilter/vf_overlay_subs.c | 1173 +++++++++++++++++++++
libavfilter/vf_overlay_subs.h | 88 ++
libavutil/frame.c | 74 +-
libavutil/frame.h | 39 +-
libavutil/version.h | 2 +-
tests/ref/fate/filter-overlay-dvdsub-2397 | 179 ++--
tests/ref/fate/sub-dvb | 162 +--
tests/ref/fate/sub2video | 49 +-
tests/ref/fate/sub2video_basic | 178 ++--
tests/ref/fate/sub2video_time_limited | 4 +-
30 files changed, 2743 insertions(+), 510 deletions(-)
create mode 100644 libavfilter/sf_sleet.c
create mode 100644 libavfilter/subtitles.c
create mode 100644 libavfilter/subtitles.h
create mode 100644 libavfilter/svf_sub2video.c
create mode 100644 libavfilter/vf_overlay_subs.c
create mode 100644 libavfilter/vf_overlay_subs.h
--
2.28.0.windows.1
More information about the ffmpeg-devel
mailing list