[FFmpeg-devel] [PATCH 2/3] avcodec/mpegvideo_enc: Limit bitrate tolerance to the representable

[James Almer]

On 5/30/2021 3:02 PM, Michael Niedermayer wrote:
> Fixes: error: 1.66789e+11 is outside the range of representable values of type 'int'
> Fixes: Ticket8201
> 
> Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> ---
>   libavcodec/mpegvideo_enc.c | 6 +++++-
>   1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/libavcodec/mpegvideo_enc.c b/libavcodec/mpegvideo_enc.c
> index c01488f483..13618394a5 100644
> --- a/libavcodec/mpegvideo_enc.c
> +++ b/libavcodec/mpegvideo_enc.c
> @@ -455,9 +455,13 @@ av_cold int ff_mpv_encode_init(AVCodecContext *avctx)
>       if (!s->fixed_qscale &&
>           avctx->bit_rate * av_q2d(avctx->time_base) >
>               avctx->bit_rate_tolerance) {
> +        double nbt = avctx->bit_rate * av_q2d(avctx->time_base) * 5;
>           av_log(avctx, AV_LOG_WARNING,
>                  "bitrate tolerance %d too small for bitrate %"PRId64", overriding\n", avctx->bit_rate_tolerance, avctx->bit_rate);
> -        avctx->bit_rate_tolerance = 5 * avctx->bit_rate * av_q2d(avctx->time_base);
> +        if (nbt <= INT_MAX) {
> +            avctx->bit_rate_tolerance = nbt;
> +        } else
> +            avctx->bit_rate_tolerance = INT_MAX;

Maybe bit_rate_tolerance can be made an int64_t? We have done that with 
all bitrate fields in AVCodecContext and similar structs.
We're still in the open ABI period, so it can be done right now.

>       }
>   
>       if (avctx->rc_max_rate &&
>