[FFmpeg-devel] Status and Plans for Subtitle Filters

[Nicolas George]

Soft Works (12020-02-14):
> I am looking for some guidance regarding future plans about processing
> subtitle streams in filter graphs.
> 
> Please correct me where I'm wrong - this is the situation as I've
> understood it so far:
> 
> - Currently, ffmpeg filter graphs do not support processing subtitle
> streams
> - This is why filters like 'subtitles' and 'ass' need to open, read
> and parse the media file a second time  in parallel instead of just
> taking the subtitle events from ffmpeg'demuxing
> - For graphical subtitles, there exists the so-called 'sub2video'
> workaround which is injecting the the graphical subtitle overlay
> images into the video filtergraph by declaring them as video
> - The 'sub2video' was meant to exist until filtering would support
> subtitle streams
> - A while ago, Clement Boesch submitted a patch for adding subtitle
> streams to filtergraph processing
> (https://patchwork.ffmpeg.org/project/ffmpeg/patch/20161102220934.26010-1-u@pkh.me/)
> - I read through all the discussion about it, but nothing happened
> afterwards and I couldn't find any indication about why it didn't get
> merged

This looks accurate.

> I'm asking because I'm intending to implement a subtitle filter that
> operates on in-stream data rather than a separate input and that will
> render text on transparent frames for later overlay.
> 
> The possible options that I have identified so far for creating that
> kind of "subtitle rendering filter" would be:
> 
> - Create a video source filter and implement some hack to get the
> subtitle data from the decoder to that filter - or...
> - Jump on the sub2video implementation and extend it to render overlay
> images  in case of text subtitles - or...
> - check out the situation with regards to adding subtitle filter
> support in ffmpeg and ask about plans for this
> 
> That's where I stand right now. Does  it even make sense, the way I
> summarized it?`

I don't speak for the project as a whole, but I am quite confident that
"some hack" would be accepted if and only if it is generic enough to be
useful for many people, not just your use case. And tweaks to sub2video
are "some hack" too.

If you want to implement real support for subtitles, that would be
greatly appreciated, but I have to warn you it is a very difficult and
intensive task. Otherwise it would have already been done. I can
summarize where the difficulty resides:

- lavfi works with AVFrame, therefore subtitles need to be encoded into
  AVFrame. This is the work of Clément that you found; he only started
  on the rest.

- lavfi is not ready to have a third media type: there are parts that
  strongly assume audio or video, and parts that merge the audio and
  video case but cannot handle a different type.

- The utility filters that only work on metadata, like setpts, need to
  be ported ported to the new media type. We don't want too much code
  duplication, a more elegant approach needs to be found. Possibly make
  the media type part of a first round of format negotiation.

- We need to decide the format negotiation for filters. Do we
  automatically insert text→bitmap renderer or does it need to be
  explicit?

- Subtitles streams are sparse, lavfi is not designed for that, and it
  is a problem when subtitles are interleaved with audio or video. The
  best way to solve it is dummy heartbeat frames. They can be
  automatically generated by the framework with an API to connect two
  buffersrc: when a video frame is added on one, generate a heartbeat
  subtitle frame on the other if necessary.

Feel free to ask details on any of these points: I do not have the
courage to start working on it for now, but I have thought about it.

Regards,

-- 
  Nicolas George
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20200217/bf139700/attachment.sig>