[FFmpeg-devel] [RFC] Format of packets for text subtitles

[Clément Bœsch]

On Tue, Jun 05, 2012 at 10:36:02PM +0200, Nicolas George wrote:
> Hi.
> 
> The purpose of this mail is to serve as a template for a specification for
> the format of AVPackets for the various text subtitles codecs.
> 
> In other words, it aims at defining the duties of the demuxers and muxers
> capable of handling text subtitles. Hopefully, if all demuxers and muxers
> follow this specification, -scodec copy should work.
> 
> The rationale for most suggestions here is: avoid special cases in muxers
> and demuxers that are not specifically for text subtitles formats.
> 

Good initiative...

> (As far as I know, Matroska is by far the most widely used format that
> supports various text subtitles styles, and the storage seems pretty sane,
> so the previous condition is mostly synonym with "do like in Matroska".)
> 
> Clément, you seem to know a lot about subtitles formats, I would appreciate
> if you comment on the relevance and correctness of what follow.
> 

Nop sorry I really don't know a lot about subtitles :)

AFAIK, Matroska is indeed an appropriate format to save subtitles; you can
put subrip, ASS, vobsub, ...

You might want to look at
http://gpac.wp.mines-telecom.fr/mp4box/ttxt-format-documentation/ though
(MP4 related).

> 
> Subtitle start time and duration are stored in AVPacket.pts and
> AVPacket.duration, in AVStream.time_base units.
> 

About timing, the DTS is also generally set; maybe it can be useful for
reordering subtitles?

(Also, in timing related stuff, we must stay aware of the karaoke)

Note: what to do about unrelated information such as comments? JACOSub for
instance can be filled with random comments all over the file. ATM I'm
dropping them (because it's a pain to handle). Should we "standardize"
the way of handling this?

> (Possible problem: ASS subtitles overlap and can start at the same time. The
> Matroska demuxer merges ASS packet with the same start time. OTOH, the text
> ASS demuxer does not, so it must not be a really big problem.)
> 
> All subtitle packets are flagged as keyframes.
> 

Demuxers must do it? In what case wouldn't you put them?

> Now for the various codecs:
> 
> CODEC_ID_TEXT
> 
>   AVCodec.extradata empty.
> 
>   AVPacket.data contains the bare text (no timestamps), in UTF-8, with line
>   separators as either LF or CRLF and no final newline.
> 
>   (Problem: OGM often contains text subtitles in legacy encoding without
>   encoding specification. I guess a demuxer option should be used.)
> 

Should we make an "input subtitle encoding" option available for all
subtitles format at a lavf top level? (available in with CONFIG_ICONV)

> CODEC_ID_SRT
> 
>   Same as CODEC_ID_TEXT, but the text can contain pseudo-HTML markup.
> 

Dropping the timing information from the packet means we might not be able
to reconstruct it exactly based on the pts+duration, but I'm not sure
that's really a problem.

Also, I would actually have a CODEC_ID_SRT and a CODEC_ID_SUBRIP:
CODEC_ID_SRT would contain the timing information + markup data, and
CODEC_ID_SUBRIP only the markup. Note that for instance the "SRT" format
can have some extra coordinates data mixed with the timing of each
event...
 
> CODEC_ID_SSA
> 
>   AVCodec.extradata contains the declarative sections, down to and including
>   the Format specification of the Events section, as plain text. The
>   Events/Format specification must be in standard order.
> 
>   AVPacket.data contains the text after the "Dialogue:" declaration, except
>   the Start and End field, and with a ReadOrder field (an integer, starting
>   at 0, that codes the order of the event in the ASS text file, which can be
>   not chronological).
> 

Just like CODEC_ID_SRT and CODEC_ID_SUBRIP, I'd propose a CODEC_ID_ASS and
a CODEC_ID_SSA (respectively file and muxed version in ASS, so the one
with timing and the other one without).

All of this is just a suggestion again, I didn't think much about it yet.

[...]

PS: sorry for the late reply...

-- 
Clément B.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20120607/103e6bca/attachment.asc>