[FFmpeg-devel] [PATCH] avcodec/libaomenc: fix use after free of sidedata

[epirat07 at gmail.com]

On 27 Nov 2024, at 14:35, Marvin Scholz wrote:

> Move the ff_dovi_configure to happen before ff_encode_add_cpb_side_data
> as the former reallocates the sidedata so the pointer returned from
> ff_encode_add_cpb_side_data would be invalid.
> ---
>  libavcodec/libaomenc.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/libavcodec/libaomenc.c b/libavcodec/libaomenc.c
> index 0f7571ee7a1..15e244d6d6a 100644
> --- a/libavcodec/libaomenc.c
> +++ b/libavcodec/libaomenc.c
> @@ -987,18 +987,18 @@ static av_cold int aom_init(AVCodecContext *avctx,
>                   (unsigned char*)1);
>
>      if (codec_caps & AOM_CODEC_CAP_HIGHBITDEPTH)
>          ctx->rawimg.bit_depth = enccfg.g_bit_depth;
>
> -    cpb_props = ff_encode_add_cpb_side_data(avctx);
> -    if (!cpb_props)
> -        return AVERROR(ENOMEM);
> -
>      ctx->dovi.logctx = avctx;
>      if ((res = ff_dovi_configure(&ctx->dovi, avctx)) < 0)
>          return res;
>
> +    cpb_props = ff_encode_add_cpb_side_data(avctx);
> +    if (!cpb_props)
> +        return AVERROR(ENOMEM);
> +
>      if (avctx->flags & AV_CODEC_FLAG_GLOBAL_HEADER) {
>          const AVBitStreamFilter *filter = av_bsf_get_by_name("extract_extradata");
>          int ret;
>
>          if (!filter) {
>
> base-commit: 78c4d6c136e10222a0b0ddff639c836f295a9029
> -- 
> 2.47.0

This patch is unnecessary if „[PATCH] avcodec/dovi_rpu: make ff_dovi_configure_ext()
take an AVCodecContext as input argument“ is merged as then it would not reallocate
the sidedata anymore.