[FFmpeg-devel] [PATCH v12 1/8] libavcodec/decode.c: intercept `AV_PKT_DATA_METADATA_UPDATE` packet extra data, attach them to the next decoded frame.

[Romain Beauxis]

Le dim. 20 avr. 2025 à 15:08, Michael Niedermayer <michael at niedermayer.cc>
a écrit :
>
> On Tue, Apr 15, 2025 at 05:22:29PM -0500, Romain Beauxis wrote:
> > ---
> >  libavcodec/decode.c | 19 +++++++++++++++++++
> >  1 file changed, 19 insertions(+)
> >
> > diff --git a/libavcodec/decode.c b/libavcodec/decode.c
> > index fca0c7ff58..06d899a9dd 100644
> > --- a/libavcodec/decode.c
> > +++ b/libavcodec/decode.c
> > @@ -97,6 +97,8 @@ typedef struct DecodeContext {
> >      int lcevc_frame;
> >      int width;
> >      int height;
> > +
> > +    AVDictionary *pending_metadata;
> >  } DecodeContext;
> >
> >  static DecodeContext *decode_ctx(AVCodecInternal *avci)
> > @@ -702,6 +704,8 @@ int attribute_align_arg
avcodec_send_packet(AVCodecContext *avctx, const AVPacke
> >  {
> >      AVCodecInternal *avci = avctx->internal;
> >      DecodeContext     *dc = decode_ctx(avci);
> > +    const uint8_t *side_metadata;
> > +    size_t size;
> >      int ret;
> >
> >      if (!avcodec_is_open(avctx) || !av_codec_is_decoder(avctx->codec))
> > @@ -719,6 +723,14 @@ int attribute_align_arg
avcodec_send_packet(AVCodecContext *avctx, const AVPacke
> >          ret = av_packet_ref(avci->buffer_pkt, avpkt);
> >          if (ret < 0)
> >              return ret;
> > +
> > +        side_metadata = av_packet_get_side_data(avpkt,
AV_PKT_DATA_METADATA_UPDATE, &size);
> > +        if (side_metadata) {
> > +            av_dict_free(&dc->pending_metadata);
> > +            ret = av_packet_unpack_dictionary(side_metadata, size,
&dc->pending_metadata);
> > +            if (ret < 0)
> > +                return ret;
> > +        }
> >      } else
> >          dc->draining_started = 1;
> >
> > @@ -788,6 +800,7 @@ fail:
> >  int ff_decode_receive_frame(AVCodecContext *avctx, AVFrame *frame)
> >  {
> >      AVCodecInternal *avci = avctx->internal;
> > +    DecodeContext     *dc = decode_ctx(avci);
> >      int ret;
> >
> >      if (avci->buffer_frame->buf[0]) {
> > @@ -810,6 +823,11 @@ int ff_decode_receive_frame(AVCodecContext *avctx,
AVFrame *frame)
> >
> >      avctx->frame_num++;
> >
> > +    if (dc->pending_metadata) {
> > +        av_dict_copy(&frame->metadata, dc->pending_metadata,
AV_DICT_APPEND);
> > +        av_dict_free(&dc->pending_metadata);
> > +    }
> > +
> >      return 0;
> >  fail:
> >      av_frame_unref(frame);
> > @@ -2220,4 +2238,5 @@ void ff_decode_internal_uninit(AVCodecContext
*avctx)
> >      DecodeContext *dc = decode_ctx(avci);
> >
> >      av_refstruct_unref(&dc->lcevc);
> > +    av_dict_free(&dc->pending_metadata);
> >  }
> > --
> > 2.39.5 (Apple Git-154)
>
> My issue with this is the same as it was previously
> Please correct me if iam wrong (it is very possible that iam wrong on
> some assumtations here)
>
> the decoder is variable delay (exact delay depends on cpu cores and luck)
> on the input we have some side data from an AVPacket
> then some packets will result in nothing to come out (because maybe they
are damaged
> or the users asked non key frames to be skiped or whatever)
> some packets will decode into AVFrames in one of several threads
> and these AVFrames then come back out
>
> how can code that seems to ignore this be correct ?
>
> Shouldnt the data be associated with the packet  go through the
> decoder with it ?
> Or is this data not associated with the packet in that sense ?

Yeah that makes sense. How about the following:
* Keep metadata's associated PTS
* Only attach a metadata to an outgoing frame with the same PTS
* Keep the latest pair of PTS/metadata in the decoder, erasing existing one
when a new one comes through.

This is a step closer to the previous tree implementation but prevents
issues with accumulating metadata.

> thx
>
>
> [...]
>
> --
> Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
>
> The educated differ from the uneducated as much as the living from the
> dead. -- Aristotle
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".