[FFmpeg-devel] Discussion: How to extend the AVFrame to hold caption data?

[Andrey Turkin]

AVFrame already has some support for embedded 608/708 subtitles in a form
of side data (AV_FRAME_DATA_A53_CC tag), and some decoders/encoders already
can handle that.
Main issue with this approach is that video frames can be modified, removed
or duplicated (mainly by video filters but also by ffmpeg's video sync
code).

2016-09-04 6:14 GMT+03:00 Jonathan Campbell <jonathan at impactstudiopro.com>:

> While I'm finishing up the CNG patch, I'd like to start developing another
> feature that I think FFMPEG should be able to at least expose through the
> API.
>
> I'm aware that FFMPEG supports subtitle tracks, but as far as I know,
> doesn't support caption data embedded in the video stream itself.
>
> Some scenarios especially in broadcast TV transmit MPEG-2 or H.264 video
> with caption data embedded in the encoded frame. I would like to write a
> patch that can read the caption data embedded in MPEG-2 user packets and
> expose it in the AVFrame.
>
> There are two common forms I work with in software: One carries EIA 608
> closed caption packets per keyframe (typically seen on DVDs here in
> America), the other carries CEA 708 closed caption data (Caption
> Description Packets) every frame (or every I/P frame) with a side channel
> for 608 (typically seen in terrestrial HDTV broadcast here in America). How
> could I modify the MPEG-2 decoder and/or bitstream parser to read and
> expose this data through the API for software that needs it? I'm also
> interested in modding the MPEG-2 encoder to take this data through the API
> and encode it into the frame. If done right, the extension could allow
> FFMPEG to transcode DVD or HDTV broadcasts while keeping the caption data
> intact.
>
> Similar standards exist to carry caption data packets in H.264 (as SEI
> packets) and DV video (as DIF packets).
>
> Jonathan Campbell
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>