[FFmpeg-devel] J2K in HEIF was: [RFC]avformat: introduce AVStreamGroup

[Tomas Härdin]

ons 2023-09-13 klockan 07:33 -0700 skrev Pierre-Anthony Lemieux:
> On Wed, Sep 13, 2023 at 2:35 AM Tomas Härdin <git at haerdin.se> wrote:
> > 
> > ons 2023-09-06 klockan 16:16 -0300 skrev James Almer:
> > > On 9/6/2023 2:53 PM, Tomas Härdin wrote:
> > > > ons 2023-09-06 klockan 11:38 -0300 skrev James Almer:
> > > > > Signed-off-by: James Almer <jamrial at gmail.com>
> > > > > ---
> > > > > This is an initial proof of concept for AVStream groups,
> > > > > something
> > > > > that's
> > > > > needed for quite a few existing and upcoming formats that
> > > > > lavf
> > > > > has no
> > > > > way to
> > > > > currently export. Said formats define a single video or audio
> > > > > stream
> > > > > composed
> > > > > by merging several individualy multiplexed streams within a
> > > > > media
> > > > > file.
> > > > > This is the case of HEIF, a format defining a tiled image
> > > > > where
> > > > > each
> > > > > tile is a
> > > > > separate image (either hevc, av1, etc) all of which need to
> > > > > be
> > > > > decoded
> > > > > individualy and then stitched together for presentation using
> > > > > container level
> > > > > information;
> > > > 
> > > > I remember this blocking HEIF as a GSoC project. Honestly the
> > > > way
> > > > that
> > > > format is designed is immensely horrible.
> > > > 
> > > > > MPEG-TS programs, currently exported as
> > > > > AVProgram, which this new general purpose API would replace.
> > > > 
> > > > I can foresee this being a nuisance for users accustomed to
> > > > AVProgram.
> > > > Also this feature borders on NLE territory. Not necessarily a
> > > > bad
> > > > thing, but FFmpeg is overall poorly architectured for NLE
> > > > stuff. I
> > > > believe I raised this issue back when lavfi was proposed, it
> > > > being
> > > > wholly unsuitable for NLE work.
> > > > 
> > > > 
> > > > > +typedef struct AVStreamGroup {
> > > > > +    /**
> > > > > +     * A class for @ref avoptions. Set on stream creation.
> > > > > +     */
> > > > > +    const AVClass *av_class;
> > > > > +
> > > > > +    /**
> > > > > +     * Group index in AVFormatContext.
> > > > > +     */
> > > > > +    int index;
> > > > > +
> > > > > +    /**
> > > > > +     * Format-specific group ID.
> > > > > +     * decoding: set by libavformat
> > > > > +     * encoding: set by the user, replaced by libavformat if
> > > > > left
> > > > > unset
> > > > > +     */
> > > > > +    int id;
> > > > > +
> > > > > +    /**
> > > > > +     * Codec parameters associated with this stream group.
> > > > > Allocated
> > > > > and freed
> > > > > +     * by libavformat in avformat_new_stream_group() and
> > > > > avformat_free_context()
> > > > > +     * respectively.
> > > > > +     *
> > > > > +     * - demuxing: filled by libavformat on stream group
> > > > > creation or
> > > > > in
> > > > > +     *             avformat_find_stream_info()
> > > > > +     * - muxing: filled by the caller before
> > > > > avformat_write_header()
> > > > > +     */
> > > > > +    AVCodecParameters *codecpar;
> > > > > +
> > > > > +    void *priv_data;
> > > > > +
> > > > > +    /**
> > > > > +     * Number of elements in AVStreamGroup.stream_index.
> > > > > +     *
> > > > > +     * Set by av_stream_group_add_stream() and
> > > > > av_stream_group_new_stream(), must not
> > > > > +     * be modified by any other code.
> > > > > +     */
> > > > > +    int nb_stream_indexes;
> > > > > +
> > > > > +    /**
> > > > > +     * A list of indexes of streams in the group. New
> > > > > entries
> > > > > are
> > > > > created with
> > > > > +     * av_stream_group_add_stream() and
> > > > > av_stream_group_new_stream().
> > > > > +     *
> > > > > +     * - demuxing: entries are created by libavformat in
> > > > > avformat_open_input().
> > > > > +     *             If AVFMTCTX_NOHEADER is set in ctx_flags,
> > > > > then
> > > > > new entries may also
> > > > > +     *             appear in av_read_frame().
> > > > > +     * - muxing: entries are created by the user before
> > > > > avformat_write_header().
> > > > > +     *
> > > > > +     * Freed by libavformat in avformat_free_context().
> > > > > +     */
> > > > > +    int *stream_index;
> > > > > +} AVStreamGroup;
> > > > 
> > > > I see no provisions for attaching metadata, for example HEIF
> > > > stitching.
> > > > Putting it in coderpar seems wrong, since it is container-level
> > > > metadata. We could just have an HEIF specific struct as
> > > > container
> > > > metadata.
> > > 
> > > The doxy for AVCodecParameters says "This struct describes the
> > > properties of an encoded stream.", so It's not about container
> > > level
> > > props.
> > 
> > It *is* container level props. The underlying codecs have no
> > concept of
> > this kind of stitching. The closest you're going to get is tiles in
> > JPEG2000, but I doubt HEIF support JPEG2000.
> 
> Just an FYI.
> 
> HEIF supports JPEG 2000:
> 
> https://www.itu.int/rec/T-REC-T.815/en
> 
> One implementation:
> 
> https://github.com/strukturag/libheif/pull/874

Cursed

/Tomas