[FFmpeg-devel] [PATCH v11 1/8] libavcodec/decode.c: intercept `AV_PKT_DATA_METADATA_UPDATE` packet extra data, attach them to the next decoded frame with the same PTS.

[Michael Niedermayer]

On Wed, Apr 09, 2025 at 09:25:56PM -0500, Romain Beauxis wrote:
> Le mer. 9 avr. 2025 à 20:12, Michael Niedermayer
> <michael at niedermayer.cc> a écrit :
> >
> > On Fri, Apr 04, 2025 at 04:14:44PM -0500, Romain Beauxis wrote:
> > > ---
> > >  libavcodec/decode.c | 130 ++++++++++++++++++++++++++++++++++++++++++++
> > >  1 file changed, 130 insertions(+)
> > >
> > > diff --git a/libavcodec/decode.c b/libavcodec/decode.c
> > > index fca0c7ff58..39d054bdea 100644
> > > --- a/libavcodec/decode.c
> > > +++ b/libavcodec/decode.c
> > [...]
> > > @@ -702,6 +809,8 @@ int attribute_align_arg avcodec_send_packet(AVCodecContext *avctx, const AVPacke
> > >  {
> > >      AVCodecInternal *avci = avctx->internal;
> > >      DecodeContext     *dc = decode_ctx(avci);
> > > +    const uint8_t *side_metadata;
> > > +    size_t size;
> > >      int ret;
> > >
> > >      if (!avcodec_is_open(avctx) || !av_codec_is_decoder(avctx->codec))
> > > @@ -719,6 +828,14 @@ int attribute_align_arg avcodec_send_packet(AVCodecContext *avctx, const AVPacke
> > >          ret = av_packet_ref(avci->buffer_pkt, avpkt);
> > >          if (ret < 0)
> > >              return ret;
> > > +
> > > +        side_metadata = av_packet_get_side_data(avpkt, AV_PKT_DATA_METADATA_UPDATE, &size);
> >
> >
> > > +        if (avpkt->pts != AV_NOPTS_VALUE && side_metadata) {
> > > +            ret = insert_pending_metadata(&dc->pending_metadata, avpkt->pts,
> > > +                                          side_metadata, size);
> > > +            if (ret < 0)
> > > +                return ret;
> >
> > Is the tree needed and a FIFO not enough ?
> 
> I believe so.
> 
> There could be scenarios where the DTS are submitted out of order and
> we'd still want the metadata to be attached to the frame it was
> intended for.
> 
> In fact, I did this change after you suggested such a scenario:
> 
> >> Can you describe a scenario that you're thinking about?
> 
> > The users feeds several packets into a multi threaded decoder
> > and then depending on the threads and luck sooner or later
> > one frame comes out.
> >
> > Passing some data in a way that disregards this, feels wrong
> > Hypothetically there also could be a 2nd AV_PKT_DATA_METADATA_UPDATE
> > going in before the frame corresponding to the first comes out
> > but i may be missing something
> 
> Source: https://ffmpeg.org/pipermail/ffmpeg-devel/2025-March/340948.html

What i was thinkig of is the user passing in multiple
AV_PKT_DATA_METADATA_UPDATE in order
internally these could decode in parallel, but what goes
back to the user is in order again. Now there is frame reordering
but i assumed that metadata updates would not be at that level.
And this would rather be concatenated streams. But i dont know ...

Also if you want to keep the tree code, then error handling may need
to be improved.
Consider that a frame with metadata goes in but decoding has an error
and this frame never comes out.

iam not sure how you want to free metadata for damaged frames that are never
output without assumtations on the ordering

thx

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Republics decline into democracies and democracies degenerate into
despotisms. -- Aristotle
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20250412/8f9f1f78/attachment.sig>