[FFmpeg-devel] [PATCH] Add flag to drop non-forced subtitle

[Oliver Fromme]

Clément B½sch wrote:
 > On Thu, May 22, 2014 at 07:39:11PM +0200, Oliver Fromme wrote:
 > > Hello,
 > > 
 > > As I mentioned earlier this week, I'm trying to improve support
 > > for forced and partially forced subtitles.  The patch below is
 > > the next step.
 > > 
 > > The patch changes four files.  I'll explain in detail what the
 > > changes are good for:
 > > 
 > > libavcodec/avcodec.h -- Introduce a new codec flag for subtitle
 > > encoders, it's called CODEC_FLAG_FORCED_SUBS.  When set, this
 > > flag indicates that the encoder should write only such subtitle
 > > frames that are marked as forced.  Frames that are *not* forced
 > > will be ignored by the encoder.
 > > 
 > 
 > > By the way, is there a rule for allocating bit masks for new
 > > flags?  I just took the first unused one, that is 0x0080.
 > > 
 > 
 > Yes, we generally introduce a gap between the last added one and the new
 > one, unless the last one added was FFmpeg specific in order to keep
 > compatibility with the fork Libav.

Oh, ok ...  So what value should be used for my patch?
I'm a bit confused; the existing constants don't even seem
to be sorted numerically.  Or should I even use flags2
instead of flags?

 > > libavcodec/options_table.h -- Add an option so the new flag can
 > > be set on the command line like this:  -flags +forced_subs
 > > 
 > > Before I continue, I need to explain some background:  A single
 > > subtitle frame can consist of multiple rectangles, each one has
 > > its own forced flag.  PGS (i.e. the Blu-Ray subtitle format)
 > > supports multiple rectangles, but VOBSUB (a.k.a. dvdsub) does
 > > not.  So, if we have a subtitle frame with multiple rectangles,
 > > we have to merge them into one rectangle when encoding to dvdsub.
 > > The dvdsubenc encoder already does that, but it doesn't respect
 > > the forced flag yet.  Now comes my patch ...
 > > 
 > > libavcodec/dvdsubenc.c -- If CODEC_FLAG_FORCED_SUBS is set and
 > > there are multiple rectangles, we need to figure out which ones
 > > are forced and which ones are not, and then merge only the ones
 > > that are forced.  The new variable "first" contains the index
 > > of the first rectangle that needs to be merged (otherwise it
 > > contains 0, so the behaviour is unchanged if the codec flag is
 > > not set).
 > > 
 > > In all the following loops that iterate over the rectangles,
 > > any rectangles that are non-forced are skipped when the codec
 > > flag is set.  That's all.
 > > 
 > 
 > > Unfortunately, there's a special case that cannot be handled
 > > here in the encoder:  When *all* rectangles of this subtitle
 > > are non-forced (and the codec flag is set).  In this case we
 > > would have to skip all rectangles and produce an empty frame.
 > > This is not possible in the encoder ...  The API expects that
 > > the codec's encode() function produces a frame for every input
 > > frame.  Therefore, this case has to be handled at a higher
 > > level:
 > 
 > That sounds like something we will want in the next subtitles encoding
 > API?

I'm not sure.  What would really be useful is a filter
mechanism for subtitles, similar to -filter and -filter_complex
for video tracks.

 > What happen if you actually do it?

First, the frame counter is increased by one, so statistics
will be wrong.  Then the write_packet function is called
(from my head, I don't have the source code in front of me
right now).  I think this function isn't able to handle the
case of zero data size correctly; it'll still try to emit
a packet.

 > > ffmpeg.c -- In the do_subtitle_out() function, we check if
 > > CODEC_FLAG_FORCED_SUBS is set.  If it is, we check if *all*
 > > of the rectangles of the current subtitle frame are non-forced.
 > > If that's the case, we return right away, not calling the
 > > encoder at all, not calling write_frame(), not increasing the
 > > frame counter etc.
 > > 
 > > You can test the new functionality with the file that I had
 > > uploaded on May 18th, called subt-forced-test.mkv.  It
 > > contains one dvdsub subtitle track that has 991 frames, 2 of
 > > which are forced.
 > > For example, the following command could be used to "split"
 > > the subtitles in two output files:  The first one contains
 > > all of them, the second one contains only the forced frames:
 > > 
 > >     $ ffmpeg -i subt-forced-test.mkv \
 > >             -codec:s dvdsub test1.mkv \
 > >             -codec:s dvdsub -flags:s +forced_subs test2.mkv
 > > 
 > >     $ mkvextract tracks test1.mkv 0:test1.idx
 > >     $ mkvextract tracks test2.mkv 0:test2.idx
 > >     $ grep -c timestamp test[12].idx
 > >     test1.idx:991
 > >     test2.idx:2
 > 
 > Just curious, what is this flag used for generally? Some forced
 > multi-language text annotation for the movie that probably anyone wants to
 > see?

When a movie is dubbed (e.g. German), normal subtitles are
not required, so the user disables subtitles on his player
or playback device.  But there are sometimes textual (i.e.
non-spoken) information in the video that is not dubbed,
things like "3 months later", the text on signs, newspaper
headlines and so on.  Subtitles are used to translate them.
Since these subtitles need to be displayed no matter if
normal subtitles are enabled or not, they are called
"forced".

Another case is when someone speaks Klingon or Elvish, and
forced subtitles are used to translate that to English
(or German or whatever).

Basecally there are three ways to put forced subtitles in
a movie:

1. "Hardcode" them into the video track.  The obvious
disadvantage is that you can't do this for multiple
languages.

2. Create a separate subtitle track and mark the whole
track as forced, if supported by the container format
(e.g. MKV).

3. Since the forced subtitles are usually a subset of the
normal subtitles, use the normal subtitle track and mark
just some of the subtitle frames inside that track as
forced, if supported by the subtitle format (e.g PGS and
dvdsub).

My patch handles the third case.  For example, it can be
used to create a separate track, i.e. convert to the
second or first case.

 > > --- ffmpeg/libavcodec/avcodec.h.orig	2014-05-13 19:20:05.000000000 +0200
 > > +++ ffmpeg/libavcodec/avcodec.h	2014-05-21 18:52:28.000000000 +0200
 > 
 > Ideally if you could do a git commit and git format-patch -1 and send the
 > result that would be great. It will include your authorship, a commit
 > message and a description, so that's way simpler for us.

Unfortunately I have *zero* knowledge about git (I'm familiar
with perforce, cvs and svn, because these are the ones that
I normally have to use).

Best regards
   Oliver

-- 
``We are all but compressed light'' (Albert Einstein)