[FFmpeg-devel] [PATCH] [RFC]avformat: introduce AVStreamGroup

[James Almer]

On 9/6/2023 2:53 PM, Tomas Härdin wrote:
> ons 2023-09-06 klockan 11:38 -0300 skrev James Almer:
>> Signed-off-by: James Almer <jamrial at gmail.com>
>> ---
>> This is an initial proof of concept for AVStream groups, something
>> that's
>> needed for quite a few existing and upcoming formats that lavf has no
>> way to
>> currently export. Said formats define a single video or audio stream
>> composed
>> by merging several individualy multiplexed streams within a media
>> file.
>> This is the case of HEIF, a format defining a tiled image where each
>> tile is a
>> separate image (either hevc, av1, etc) all of which need to be
>> decoded
>> individualy and then stitched together for presentation using
>> container level
>> information;
> 
> I remember this blocking HEIF as a GSoC project. Honestly the way that
> format is designed is immensely horrible.
> 
>> MPEG-TS programs, currently exported as
>> AVProgram, which this new general purpose API would replace.
> 
> I can foresee this being a nuisance for users accustomed to AVProgram.
> Also this feature borders on NLE territory. Not necessarily a bad
> thing, but FFmpeg is overall poorly architectured for NLE stuff. I
> believe I raised this issue back when lavfi was proposed, it being
> wholly unsuitable for NLE work.
> 
> 
>> +typedef struct AVStreamGroup {
>> +    /**
>> +     * A class for @ref avoptions. Set on stream creation.
>> +     */
>> +    const AVClass *av_class;
>> +
>> +    /**
>> +     * Group index in AVFormatContext.
>> +     */
>> +    int index;
>> +
>> +    /**
>> +     * Format-specific group ID.
>> +     * decoding: set by libavformat
>> +     * encoding: set by the user, replaced by libavformat if left
>> unset
>> +     */
>> +    int id;
>> +
>> +    /**
>> +     * Codec parameters associated with this stream group. Allocated
>> and freed
>> +     * by libavformat in avformat_new_stream_group() and
>> avformat_free_context()
>> +     * respectively.
>> +     *
>> +     * - demuxing: filled by libavformat on stream group creation or
>> in
>> +     *             avformat_find_stream_info()
>> +     * - muxing: filled by the caller before avformat_write_header()
>> +     */
>> +    AVCodecParameters *codecpar;
>> +
>> +    void *priv_data;
>> +
>> +    /**
>> +     * Number of elements in AVStreamGroup.stream_index.
>> +     *
>> +     * Set by av_stream_group_add_stream() and
>> av_stream_group_new_stream(), must not
>> +     * be modified by any other code.
>> +     */
>> +    int nb_stream_indexes;
>> +
>> +    /**
>> +     * A list of indexes of streams in the group. New entries are
>> created with
>> +     * av_stream_group_add_stream() and
>> av_stream_group_new_stream().
>> +     *
>> +     * - demuxing: entries are created by libavformat in
>> avformat_open_input().
>> +     *             If AVFMTCTX_NOHEADER is set in ctx_flags, then
>> new entries may also
>> +     *             appear in av_read_frame().
>> +     * - muxing: entries are created by the user before
>> avformat_write_header().
>> +     *
>> +     * Freed by libavformat in avformat_free_context().
>> +     */
>> +    int *stream_index;
>> +} AVStreamGroup;
> 
> I see no provisions for attaching metadata, for example HEIF stitching.
> Putting it in coderpar seems wrong, since it is container-level
> metadata. We could just have an HEIF specific struct as container
> metadata.

The doxy for AVCodecParameters says "This struct describes the 
properties of an encoded stream.", so It's not about container level props.

Although codecpar will be used to export the merged/stitched stream 
props like dimensions and channel layout, maybe you're right about the 
metadata because there would be a clash between actual HEVC/Opus/AAC/AV1 
extradata and the HEIF/IAMF/etc specific info if both use 
codecpar.extradata, even if one will be in AVStream and the other in 
AVStreamGroup.

Maybe in side_data (Once my other set is pushed)? Defining new types for 
each kind of metadata.