[FFmpeg-devel] [RFC] Subtitle Filtering Ramp-Up

[Nicolas George]

Zach Swena (HE12025-06-03):
> The way I see it, rudeness has been present on all sides of most of the
> conflicts I have seen on the ML lately and it is way too easy to reflect
> rudeness back so yes I think everyone on here needs to get over it and work
> together politely.

How do you work politely with somebody who says “I am not insulting
people calling them crazy because they are really crazy”?

> This is why I have been trying to ask high level questions.  His old
> subtitle filtering patchset would need a lot of rework to bring to the
> current codebase so starting over isn't a bad idea.  I don't really care
> who works on or makes the commits for the code as long as the resulting
> code is clean, makes sense to other developers and accomplishes what
> everyone needs.  There are structural changes needed for what I want and it
> would be good to find a solution that allows for functionality for
> additional processing options also.

His old filtering patchet was broken beyond repair, and nothing changed.

Just to give an idea of my position on the topic: during one of the
first VDDs, possibly the first one, I co-hosted with Ubitux a
multi-hours brainstorming session on the matter of subtitles in
libavfilter. That is how much I want to keep subtitles out of
libavfilter, and that is how little time I have spent on it anticipating
problems and finding solution.

When I say that softworkz's approach is broken, I know what I am
talking about.

It is broken in three ways.

The first way is the hardest: it does not handle syncing, all the
framesync code plus the complication of subtitles being sparse. It just
feeds everything to libass and takes what comes out of it. It works when
the subtitles arrive neatly before the video frames. It just does not
work for such a simple use case as shifting the subtitles a few seconds
forward.

softworkz has refused to even test that case.

But softworkz could not even test that case, because, second flaw, the
easiest to fix: he neglected to implement all the utility filters, the
filters that operate not on the frame payload but on timestamps, side
data, other technical properties. All these filters are needed for
subtitles too. softworkz patch series do not include them, and he
refuses to implement them.

It is a little harder than it looks, because implementing these filters
by copy-pasting a third copy would not be acceptable, it must be done
by factoring the existing duplicated code. But it being a little harder
is not an excuse not to work at it.

Third flaw: no negotiation. Negotiation is a core concept of
libavfilter: have a RGB stream and a YUV-only filter? lavfi inserts the
scale filter, and it inserts it at the right place. With softworkz's
implementation: have text subtitles, expect bitmap ones, it fails
instead of inserting the rasterizer at the right place.

This series only works in the simplest of test cases. It does not handle
even one of the most obvious use cases, adjusting timing. It cannot even
be called a proof of concept.

If we want to move libavfilter fowrward properly, there are preparatory
things that need to happen. Not sexy things that will let you put your
name on a “killer feature”, which seems to be all softworkz wants, not
things that will cause randos on Twitter swoon “ffmpeg is so good!”, but
things that are necessary to do things properly.

These things are some of the things I criticized about softworkz's patch
series: refactoring the negotiation code and then the utility filters.

The issue with this is that the negotiation code is extremely fragile
and has barely any test coverage at all.

Therefore, the very first thing that needs to happen on libavfilter is
to add test coverage on the negotiation system.

That is not hard work. That is not glorious work either, in fact it is
extremely boring, which is why I never completed it.

But it needs to happen.

Just for the record, I asked softworkz, I told him: for your patch
series to move forward, we need to add testing, will you help? No help
came.

Regards,

-- 
  Nicolas George