[FFmpeg-devel] [PATCH] WebM muxer writes WebVTT subtitle track

[Clément Bœsch]

On Mon, Jun 03, 2013 at 01:16:32PM -0700, Matthew Heaney wrote:
> The Matroska muxer now allows WebVTT subtitle tracks to be written
> while in WebM muxing mode.
> 
> WebVTT subtitle tracks have four kinds: "subtitles", "captions",
> "descriptions", and "metadata". Each text track kind has a distinct
> Mastroska CodecID and track type, as described in the temporal
> metadata guidelines here:
> 
> http://wiki.webmproject.org/webm-metadata/temporal-metadata/webvtt-in-webm
> 

Maybe you could use AV_DISPOSITION_* instead?

> In ffmpeg, the WebVTT kind can be specified explicitly using the
> -metadata option to specify a mapping from key "kind" to one of the
> kind values listed above.  The kind "subtitles" is the default if no
> mapping is specified.
> 
> When the stream has codec id AV_CODEC_ID_WEBVTT, the stream packet is
> serialized per the temporal metadata guidelines cited above. The
> WebVTT cue is written as a Matroska block group. The block frame
> comprises the WebVTT cue id, followed by the cue settings, followed by
> the cue text.  (The block timestamp is synthesized from the cue
> timestamp.)
> ---
>  libavformat/matroskaenc.c | 145 ++++++++++++++++++++++++++++++++++++++++------
>  1 file changed, 127 insertions(+), 18 deletions(-)
> 

Note: All of this sounds extremely complex in comparison to the subtitles
codecs you can mux in matroska (SRT and ASS are deprecated, there is now a
single function for all of them).

> diff --git a/libavformat/matroskaenc.c b/libavformat/matroskaenc.c
> index 99d648d..77c9fbb 100644
> --- a/libavformat/matroskaenc.c
> +++ b/libavformat/matroskaenc.c
> @@ -560,6 +560,41 @@ static int mkv_write_tracks(AVFormatContext *s)
>      ebml_master tracks;
>      int i, j, ret, default_stream_exists = 0;
>  
> +    for (i = 0; i < s->nb_streams; i++) {
> +        AVStream *st = s->streams[i];
> +        AVCodecContext *codec = st->codec;
> +        AVDictionaryEntry *t;
> +

> +        if (mkv->mode != MODE_WEBM)
> +            continue;
> +

This check can be moved outside the loop.

> +        if (codec->codec_id == AV_CODEC_ID_VP8
> +            || codec->codec_id == AV_CODEC_ID_VORBIS) {
> +            continue;
> +        }
> +
> +        if (codec->codec_id != AV_CODEC_ID_WEBVTT) {
> +            av_log(s, AV_LOG_ERROR,
> +                   "Only VP8 video, Vorbis audio, and WebVTT subtitles are supported for WebM.\n");
> +            return AVERROR(EINVAL);
> +        }
> +
> +        t = av_dict_get(st->metadata, "kind", NULL, 0);
> +
> +        if (!t)
> +          continue;
> +
> +        if (!av_strcasecmp(t->value, "subtitles")
> +            || !av_strcasecmp(t->value, "captions")
> +            || !av_strcasecmp(t->value, "descriptions")
> +            || !av_strcasecmp(t->value, "metadata")) {
> +            continue;
> +        }
> +
> +        av_log(s, AV_LOG_ERROR, "Bad WebVTT kind specifier.\n");
> +        return AVERROR(EINVAL);
> +    }
> +
[...]
> +static int mkv_write_vtt_blocks(AVFormatContext *s, AVIOContext *pb, AVPacket *pkt) {
> +    MatroskaMuxContext *mkv = s->priv_data;
> +    ebml_master blockgroup;
> +    int id_size, settings_size, size;
> +    uint8_t *id, *settings;
> +    int64_t ts = mkv->tracks[pkt->stream_index].write_dts ? pkt->dts : pkt->pts;
> +    const int flags = 0;

> +    const uint8_t EOL[] = "\n";
> +

Oh really? It's separated by \n in the binary matroska muxing? If that so,
my suggestion of webvtt identifier and settings for side data was a bad
idea (or muxing them that way in webm is stupid). I was hoping those
identifier and settings were in a dedicated webm field or something, but
they are actually part of the payload, and thus any subtitles event
without settings or identifier is supposed to start with "\n\n"?...

If the webvtt muxing in webm is not a draft anymore, I believe we should
remove those side data and create those weird packets payload (and change
our decoder). Sorry for suggesting the side data in the first place.

> +    id_size = 0;
> +    id = av_packet_get_side_data(pkt, AV_PKT_DATA_WEBVTT_IDENTIFIER,
> +                                 &id_size);
> +
> +    settings_size = 0;
> +    settings = av_packet_get_side_data(pkt, AV_PKT_DATA_WEBVTT_SETTINGS,
> +                                       &settings_size);
> +
> +    size = id_size + 1 + settings_size + 1 + pkt->size;
> +
> +    av_log(s, AV_LOG_DEBUG, "Writing block at offset %" PRIu64 ", size %d, "
> +           "pts %" PRId64 ", dts %" PRId64 ", duration %d, flags %d\n",
> +           avio_tell(pb), size, pkt->pts, pkt->dts, pkt->duration, flags);
> +
> +    blockgroup = start_ebml_master(pb, MATROSKA_ID_BLOCKGROUP, mkv_blockgroup_size(size));
> +
> +    put_ebml_id(pb, MATROSKA_ID_BLOCK);
> +    put_ebml_num(pb, size+4, 0);
> +    avio_w8(pb, 0x80 | (pkt->stream_index + 1));     // this assumes stream_index is less than 126
> +    avio_wb16(pb, ts - mkv->cluster_pts);
> +    avio_w8(pb, flags);
> +    avio_write(pb, id, id_size);
> +    avio_write(pb, EOL, 1);
> +    avio_write(pb, settings, settings_size);
> +    avio_write(pb, EOL, 1);
> +    avio_write(pb, pkt->data, pkt->size);
> +
> +    put_ebml_uint(pb, MATROSKA_ID_BLOCKDURATION, pkt->duration);
> +    end_ebml_master(pb, blockgroup);
> +
> +    return pkt->duration;
> +}
> +

[...]

-- 
Clément B.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 490 bytes
Desc: not available
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20130608/8c0bd3d6/attachment.asc>