[FFmpeg-devel] [PATCH] Add flag to drop non-forced subtitle

[Oliver Fromme]

Michael Niedermayer wrote:
 > On Mon, May 26, 2014 at 12:03:35PM +0200, Oliver Fromme wrote:
 > > Michael Niedermayer wrote:
 > > > On Thu, May 22, 2014 at 07:39:11PM +0200, Oliver Fromme wrote:
 > > > > Hello,
 > > > > 
 > > > > As I mentioned earlier this week, I'm trying to improve support
 > > > > for forced and partially forced subtitles.  The patch below is
 > > > > the next step.
 > > > > 
 > > > > The patch changes four files.  I'll explain in detail what the
 > > > > changes are good for:
 > > > > 
 > > > > libavcodec/avcodec.h -- Introduce a new codec flag for subtitle
 > > > > encoders, it's called CODEC_FLAG_FORCED_SUBS.  When set, this
 > > > > flag indicates that the encoder should write only such subtitle
 > > > > frames that are marked as forced.  Frames that are *not* forced
 > > > > will be ignored by the encoder.
 > > > > 
 > > > > By the way, is there a rule for allocating bit masks for new
 > > > > flags?  I just took the first unused one, that is 0x0080.
 > > > > 
 > > > > libavcodec/options_table.h -- Add an option so the new flag can
 > > > > be set on the command line like this:  -flags +forced_subs
 > > > > 
 > > > > Before I continue, I need to explain some background:  A single
 > > > > subtitle frame can consist of multiple rectangles, each one has
 > > > > its own forced flag.  PGS (i.e. the Blu-Ray subtitle format)
 > > > > supports multiple rectangles, but VOBSUB (a.k.a. dvdsub) does
 > > > > not.  So, if we have a subtitle frame with multiple rectangles,
 > > > > we have to merge them into one rectangle when encoding to dvdsub.
 > > > > The dvdsubenc encoder already does that, but it doesn't respect
 > > > > the forced flag yet.  Now comes my patch ...
 > > > > 
 > > > > libavcodec/dvdsubenc.c -- If CODEC_FLAG_FORCED_SUBS is set and
 > > > > there are multiple rectangles, we need to figure out which ones
 > > > > are forced and which ones are not, and then merge only the ones
 > > > > that are forced.  The new variable "first" contains the index
 > > > > of the first rectangle that needs to be merged (otherwise it
 > > > > contains 0, so the behaviour is unchanged if the codec flag is
 > > > > not set).
 > > > > 
 > > > > In all the following loops that iterate over the rectangles,
 > > > > any rectangles that are non-forced are skipped when the codec
 > > > > flag is set.  That's all.
 > > > > 
 > > > > Unfortunately, there's a special case that cannot be handled
 > > > > here in the encoder:  When *all* rectangles of this subtitle
 > > > > are non-forced (and the codec flag is set).  In this case we
 > > > > would have to skip all rectangles and produce an empty frame.
 > > > > This is not possible in the encoder ...  The API expects that
 > > > > the codec's encode() function produces a frame for every input
 > > > > frame.  Therefore, this case has to be handled at a higher
 > > > > level:
 > > > > 
 > > > > ffmpeg.c -- In the do_subtitle_out() function, we check if
 > > > > CODEC_FLAG_FORCED_SUBS is set.  If it is, we check if *all*
 > > > > of the rectangles of the current subtitle frame are non-forced.
 > > > > If that's the case, we return right away, not calling the
 > > > > encoder at all, not calling write_frame(), not increasing the
 > > > > frame counter etc.
 > > > > 
 > > > > You can test the new functionality with the file that I had
 > > > > uploaded on May 18th, called subt-forced-test.mkv.  It
 > > > > contains one dvdsub subtitle track that has 991 frames, 2 of
 > > > > which are forced.
 > > > > For example, the following command could be used to "split"
 > > > > the subtitles in two output files:  The first one contains
 > > > > all of them, the second one contains only the forced frames:
 > > > > 
 > > > >     $ ffmpeg -i subt-forced-test.mkv \
 > > > >             -codec:s dvdsub test1.mkv \
 > > > >             -codec:s dvdsub -flags:s +forced_subs test2.mkv
 > > > > 
 > > > >     $ mkvextract tracks test1.mkv 0:test1.idx
 > > > >     $ mkvextract tracks test2.mkv 0:test2.idx
 > > > >     $ grep -c timestamp test[12].idx
 > > > >     test1.idx:991
 > > > >     test2.idx:2
 > > > > 
 > > > > Voila.
 > > > > 
 > > > > Best regards
 > > > >    Oliver
 > > > > 
 > > > > 
 > > > > --- ffmpeg/libavcodec/avcodec.h.orig	2014-05-13 19:20:05.000000000 +0200
 > > > > +++ ffmpeg/libavcodec/avcodec.h	2014-05-21 18:52:28.000000000 +0200
 > > > > @@ -758,6 +758,7 @@
 > > > >   */
 > > > >  #define CODEC_FLAG_MV0    0x0040
 > > > >  #endif
 > > > > +#define CODEC_FLAG_FORCED_SUBS     0x0080   ///< Encode forced subtitles only.
 > > > >  #if FF_API_INPUT_PRESERVED
 > > > >  /**
 > > > >   * @deprecated passing reference-counted frames to the encoders replaces this
 > > > > --- ffmpeg/libavcodec/options_table.h.orig	2014-05-13 19:20:05.000000000 +0200
 > > > > +++ ffmpeg/libavcodec/options_table.h	2014-05-21 18:55:15.000000000 +0200
 > > > > @@ -60,6 +60,7 @@
 > > > >  #if FF_API_MV0
 > > > >  {"mv0", "always try a mb with mv=<0,0>", 0, AV_OPT_TYPE_CONST, {.i64 = CODEC_FLAG_MV0 }, INT_MIN, INT_MAX, V|E, "flags"},
 > > > >  #endif
 > > > > +{"forced_subs", "encode forced subtitle captions only", 0, AV_OPT_TYPE_CONST, {.i64 = CODEC_FLAG_FORCED_SUBS }, INT_MIN, INT_MAX, S|E, "flags"},
 > > > >  #if FF_API_INPUT_PRESERVED
 > > > >  {"input_preserved", NULL, 0, AV_OPT_TYPE_CONST, {.i64 = CODEC_FLAG_INPUT_PRESERVED }, INT_MIN, INT_MAX, 0, "flags"},
 > > > >  #endif
 > > > 
 > > > > --- ffmpeg/libavcodec/dvdsubenc.c.orig	2014-05-16 23:35:29.000000000 +0200
 > > > > +++ ffmpeg/libavcodec/dvdsubenc.c	2014-05-21 20:03:55.000000000 +0200
 > > > 
 > > > why is this done in dvdsubenc and not utils.c for all subtitle
 > > > encoders ?
 > > 
 > > utils.c has no knowledge if the subtitle codec supports 
 > > multiple rectangles per subtitle frame.  dvdsub does not,
 > > so dvdsubenc has to merge multiple rectangles into one
 > > frame, and also handle per-rectangle forced flags.
 > 
 > i dont understand the problem you describe
 > 
 > your patch adds a flag that says "encode forced subtitle captions only"
 > so it should be possible to drop non forced rectangles before the
 > encoder in utils.c, what am i missing ?

First of all, I'm sorry for the late reply.  I was swamped with
other work and private things, and had near zero spare time.

I agree with you that it is not optimal to filter the rectangles
in the dvdsub encoder.  I'm not very happy with this either.
It would be better to do that in one place for all encoders
(even though dvdsubenc is currently ffmpeg's only encoder for
graphical subtitles).

The problem with dropping rectangles in utils.c is that there
might be multiple output streams.  The flag ("forced subtitles
only") might be set for some output streams, but not for others.
So, if we dropped some rectangles from the AVSubtitle structure
in utils.c, they will be missing from other output streams that
are encoded afterwards.

On the other hand, the dvdsub encoder has to merge rectangles
into one frame, so it has to have a loop for them anyway, so it
was very easy to add the new feature there.  Basically it's just
one "if" instruction in the loops that iterate the rectangles.
That's why I chose this solution.

If you have a better idea how to solve the problem, please let
me know.

Best regards
   Oliver


-- 
``We are all but compressed light'' (Albert Einstein)