[FFmpeg-devel] [PATCH] avcodec/nvdec: support resizing while decoding

[Carlos Ruiz]

> We don't really leverage these extra functions of NVDEC because it
> breaks many assumptions about hwaccels, which are meant to be exact
> decoders.

Yeah I understand that, and expected this kind of feedback. Do you envision
a
way for hwaccels (perhaps in the future) to support some additional level of
*hardware acceleration*, such as resizing and cropping? Let me try to
motivate
the rationale.

Think about a common use case in the AI world we live in: you receive a
bunch of
simultaneous 4k (or 1080p) incoming rtsp streams and you want to decode the
video
and pass it through some ML model. The native hevc codec doesn't support
resizing,
so you decode video at full 4k on the gpu by leveraging the nvdec hwaccel,
which
allocates something like 5-10 surfaces at 3840x2160, totalling around 250MB
of VRAM
(GPU memory), and then you have to immediately take all of those frames,
pass them
through a filterchain, scale them down to e.g. 640x360. Leaving aside the
waste of
additional CUDA cores and added latency of having to synchronize the cuda
stream twice
(once to receive the frame out of the decoder, the second to pull from the
filterchain),
do this for 50 camera streams and you'll quickly run out of GPU memory with
a GPU utilization
under 10%... Instead, my patch allows decoding at 640x360 (or whatever you
choose)
directly, allocating only 6MB of GPU VRAM (instead of 250MB at 4k), so now
you can fit
40x more video decoding streams in the same GPU card.

I understand the challenges of allowing hwaccels to resize/crop as part of
the decoding
process and how it currently breaks the API in several ways I'm of course
unfamiliar with.
However, I'd love to help support this feature in the future, since it
would add tremendous
value and enable more efficient real-time Computer Vision pipelines to
leverage the
amazing framework FFmpeg is.


> You cannot do that here. The frame size can change mid-stream, and
> this would suppress any such change.
> Additionally, if this code runs more then once, then the offset is
> applied repeatedly without actually resetting width/height.

Very good points! Indeed I'm not yet super familiar with the structure of
the hevc codec implementation, nor with the available functions to
reconfigure
a hwaccel (I know the Video Codec SDK supports this, but have never
implemented such functionality), but I see how this would break.
Would you see value in me trying to figure out a way to support
reconfiguring
the codec if the width/height/cropping changes mid-stream or would you
reject
the patch/feature regardless?

Thanks!

On Fri, Sep 20, 2024 at 7:18 AM Hendrik Leppkes <h.leppkes at gmail.com> wrote:

> On Fri, Sep 20, 2024 at 1:24 AM Carlos Ruiz <carlos.r.domin at gmail.com>
> wrote:
> >
> > Hi!
> >
> > This is my first contribution to the project so please excuse any bad
> > etiquette, I tried to read all the FAQs before posting. Would love to
> start
> > by thanking everyone for such an amazing framework you've built!
> >
> > Anyway, here's my proposed patch to support video resizing when using
> NVDEC
> > hwaccel to decode hevc video (I could look into a similar patch for h264,
> > av1, etc if this looks useful). There's a bit more context/explanation in
> > the commit description in the patch, but please let me know if the use
> case
> > isn't clear.
> >
>
> We don't really leverage these extra functions of NVDEC because it
> breaks many assumptions about hwaccels, which are meant to be exact
> decoders.
> If anything, just fudging the width/height is certainly an API
> violation and will likely not be possible as it breaks many
> assumptions in the code otherwise, see below.
>
> > ---
> >  libavcodec/hevc/hevcdec.c |  8 ++++++--
> >  libavcodec/nvdec.c        | 21 +++++++++++++++++----
> >  2 files changed, 23 insertions(+), 6 deletions(-)
> >
> > diff --git a/libavcodec/hevc/hevcdec.c b/libavcodec/hevc/hevcdec.c
> > index d915d74d22..d63fc5875f 100644
> > --- a/libavcodec/hevc/hevcdec.c
> > +++ b/libavcodec/hevc/hevcdec.c
> > @@ -351,8 +351,12 @@ static void export_stream_params(HEVCContext *s,
> const
> > HEVCSPS *sps)
> >      avctx->pix_fmt             = sps->pix_fmt;
> >      avctx->coded_width         = sps->width;
> >      avctx->coded_height        = sps->height;
> > -    avctx->width               = sps->width  - ow->left_offset -
> > ow->right_offset;
> > -    avctx->height              = sps->height - ow->top_offset  -
> > ow->bottom_offset;
> > +    if (avctx->width <= 0 || avctx->height <= 0) {
> > +        avctx->width           = sps->width;
> > +        avctx->height          = sps->height;
> > +    }
> > +    avctx->width               = avctx->width - ow->left_offset -
> ow->right_offset;
> > +    avctx->height              = avctx->height - ow->top_offset  -
> ow->bottom_offset;
>
> You cannot do that here. The frame size can change mid-stream, and
> this would suppress any such change.
> Additionally, if this code runs more then once, then the offset is
> applied repeatedly without actually resetting width/height.
>
> - Hendrik
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>