[FFmpeg-devel] [PATCH 3/3] avcodec/mpegvideo_enc: use 64bit multiplication in dct_quantize_trellis_c and dct_quantize_c

[Ramiro Polla]

Hi Marton,

On Thu, Jan 16, 2025 at 8:52 PM Marton Balint <cus at passwd.hu> wrote:
> On Tue, 7 Jan 2025, Marton Balint wrote:
> > Fixes corruption with:
> >
> > ffmpeg -t 1 -filter_complex "sine=f=21,showwaves=scale=cbrt:mode=line:colors=white:draw=full" -c:v mpeg2video -non_linear_quant 1 -qmin 1 -qmax 1 -cpuflags 0 out.mpg
> >
> > or
> >
> > ffmpeg -t 1 -filter_complex "sine=f=21,showwaves=scale=cbrt:mode=line:colors=white:draw=full" -c:v mpeg2video -non_linear_quant 1 -qmin 1 -qmax 1 -trellis 1 out.mpg
> >
>
> Ping for this.

I checked that your examples do indeed get fixed by this patch, and
also without -cpuflags 0 on x86.

It would be great if we could add more comments about the range for
the coefficients after the fdct, to make it clearer why the overflow
happened.

I wanted to do a more thorough check on the x86 code, and started
adding dct_quantize to checkasm. The idea was to also check for
overflow in multiple cases (mpeg/jpeg input range, jpeg/mpeg2/mpeg4,
mb_intra...) but I gave up after trying to split the minimum necessary
code to test dct_quantize from ff_mpv_encode_init. I will probably
look into this in the future again, unless someone else wants to do
it.

All that just to say that the patch looks good to me as-is, but more
work could be done afterwards.

Ramiro