[FFmpeg-devel] [PATCH v3 2/2] avcodec/vpp_qsv: Copy side data from input to output frame

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of Anton
> Khirnov
> Sent: Tuesday, December 7, 2021 9:04 AM
> To: ffmpeg-devel at ffmpeg.org
> Subject: Re: [FFmpeg-devel] [PATCH v3 2/2] avcodec/vpp_qsv: Copy side data
> from input to output frame
> 
> Quoting Soft Works (2021-12-03 08:58:31)
> > Signed-off-by: softworkz <softworkz at hotmail.com>
> > ---
> >  libavfilter/qsvvpp.c         |  5 +++++
> >  libavfilter/vf_overlay_qsv.c | 19 +++++++++++++++----
> >  2 files changed, 20 insertions(+), 4 deletions(-)
> >
> > diff --git a/libavfilter/qsvvpp.c b/libavfilter/qsvvpp.c
> > index d1218355c7..b291216292 100644
> > --- a/libavfilter/qsvvpp.c
> > +++ b/libavfilter/qsvvpp.c
> > @@ -849,6 +849,11 @@ int ff_qsvvpp_filter_frame(QSVVPPContext *s,
> AVFilterLink *inlink, AVFrame *picr
> >                  return AVERROR(EAGAIN);
> >              break;
> >          }
> > +
> > +        ret = av_frame_copy_side_data(out_frame->frame, in_frame->frame,
> 0);
> > +        if (ret < 0)
> > +            return ret;
> > +
> >          out_frame->frame->pts = av_rescale_q(out_frame-
> >surface.Data.TimeStamp,
> >                                               default_tb, outlink-
> >time_base);
> >
> > diff --git a/libavfilter/vf_overlay_qsv.c b/libavfilter/vf_overlay_qsv.c
> > index 7e76b39aa9..02518e020c 100644
> > --- a/libavfilter/vf_overlay_qsv.c
> > +++ b/libavfilter/vf_overlay_qsv.c
> > @@ -231,13 +231,24 @@ static int process_frame(FFFrameSync *fs)
> >  {
> >      AVFilterContext  *ctx = fs->parent;
> >      QSVOverlayContext  *s = fs->opaque;
> > +    AVFrame       *frame0 = NULL;
> >      AVFrame        *frame = NULL;
> > -    int               ret = 0, i;
> > +    int               ret = 0;
> >
> > -    for (i = 0; i < ctx->nb_inputs; i++) {
> > +    for (unsigned i = 0; i < ctx->nb_inputs; i++) {
> >          ret = ff_framesync_get_frame(fs, i, &frame, 0);
> > -        if (ret == 0)
> > -            ret = ff_qsvvpp_filter_frame(s->qsv, ctx->inputs[i], frame);
> > +
> > +        if (ret == 0) {
> > +            AVFrame *temp;
> > +
> > +            if (i == 0)
> > +                frame0 = frame;
> > +            else
> > +                ret = av_frame_copy_side_data(frame, frame0, 0);
> > +
> > +            ret = ret < 0 ? ret : ff_qsvvpp_filter_frame(s->qsv, ctx-
> >inputs[i], frame);
> 
> I don't quite understand the ownership semantics here. This function
> does not free frame, so I assume ff_qsvvpp_filter_frame() takes
> ownership of it. That would mean you're not allowed to keep a pointer to
> it and access it later, because it might have already been freed.

The filter is using framesync, which is taking care of the ownership.
ff_qsvvpp_filter_frame() clones or copies the frame, depending on case.
Other than with the normal overlay filter, the frame from input0 is
not used for output. But the regular overlay filter has established the
convention that side data from input0 is being kept at the output.

This patch is making sure that overlay_qsv behaves in the same way.
The problem is that (due to the async nature of processing in qsvvpp)
it can't always take the side data from the input0-frame, because there's
no output frame at this stage. So we copy the side data to the frame from
input1, to make sure that the "right" side data is copied (transferred) in
either case.

Thanks for looking at this,
softworkz