[FFmpeg-devel] [PATCH 4/4] avcodec/osq: avoid using too large numbers for shifts and integers in update_residue_parameter()

[Paul B Mahol]

On Fri, Sep 15, 2023 at 3:12 PM Michael Niedermayer <michael at niedermayer.cc>
wrote:

> The code should be changed to not use floats in the VLC parameters
> This patch merely fixes undefined behavior
>
> Fixes: 2.96539e+09 is outside the range of representable values of type
> 'int'
> Fixes: Assertion n>=0 && n<=32 failed at libavcodec/get_bits.h:423
> Fixes:
> 62241/clusterfuzz-testcase-minimized-ffmpeg_AV_CODEC_ID_OSQ_fuzzer-4525761925873664
>


NAK

Breaks decoding.


>
> Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> ---
>  libavcodec/osq.c | 10 +++++++---
>  1 file changed, 7 insertions(+), 3 deletions(-)
>
> diff --git a/libavcodec/osq.c b/libavcodec/osq.c
> index e2a84657fb4..e7f11691d2e 100644
> --- a/libavcodec/osq.c
> +++ b/libavcodec/osq.c
> @@ -152,11 +152,15 @@ static int update_residue_parameter(OSQChannel *cb)
>
>      sum = cb->sum;
>      x = sum / cb->count;
> -    rice_k = av_ceil_log2(x);
> +    rice_k = ceil(log2(x));
>      if (rice_k >= 30) {
> -        rice_k = floor(sum / 1.4426952 + 0.5);
> -        if (rice_k < 1)
> +        double f = floor(sum / 1.4426952 + 0.5);
> +        if (f <= 1) {
>              rice_k = 1;
> +        } else if (f >= 31) {
> +            rice_k = 31;
> +        } else
> +            rice_k = f;
>      }
>
>      return rice_k;
> --
> 2.17.1
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".
>