[FFmpeg-devel] [PATCH] vc2enc: properly promote operations to 64 bits

[Rostislav Pehlivanov]

On 12 March 2018 at 23:36, Rostislav Pehlivanov <atomnuker at gmail.com> wrote:

> On Windows machines, the UL suffix still means 32 bits.
> The only parts that need 64 bits are (1ULL << (m + 32)) and
> (t*qf + qf). Hence, use the proper ULL suffix for the former
> and just increase the type of the qf constant for the latter.
> No overflows can happen as long as these are done in 64 bits and
> the quantization table doesn't change.
>
> Signed-off-by: Rostislav Pehlivanov <atomnuker at gmail.com>
> ---
>  libavcodec/vc2enc.c | 10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)
>
> diff --git a/libavcodec/vc2enc.c b/libavcodec/vc2enc.c
> index 2e480ba8d0..d0101e01e4 100644
> --- a/libavcodec/vc2enc.c
> +++ b/libavcodec/vc2enc.c
> @@ -1163,14 +1163,14 @@ static av_cold int vc2_encode_init(AVCodecContext
> *avctx)
>          goto alloc_fail;
>
>      for (i = 0; i < 116; i++) {
> -        const uint32_t qf = ff_dirac_qscale_tab[i];
> -        const int m = av_log2(qf);
> -        const uint32_t t = (1UL << (m + 32)) / qf;
> -        const uint32_t r = (t*qf + qf) & ((1UL << 32) - 1);
> +        const uint64_t qf = ff_dirac_qscale_tab[i];
> +        const uint32_t m = av_log2(qf);
> +        const uint32_t t = (1ULL << (m + 32)) / qf;
> +        const uint32_t r = (t*qf + qf) & UINT32_MAX;
>          if (!(qf & (qf - 1))) {
>              s->qmagic_lut[i][0] = 0xFFFFFFFF;
>              s->qmagic_lut[i][1] = 0xFFFFFFFF;
> -        } else if (r <= 1UL << m) {
> +        } else if (r <= 1 << m) {
>              s->qmagic_lut[i][0] = t + 1;
>              s->qmagic_lut[i][1] = 0;
>          } else {
> --
> 2.16.2
>
>
Fixes what its meant to fix, pushed