[FFmpeg-devel] [PATCH] vc2enc_dwt: use 32 bit coefficients by default

[Rostislav Pehlivanov]

On 7 May 2016 at 18:12, Rostislav Pehlivanov <atomnuker at gmail.com> wrote:

> The problem is that with particularly complex images and especially at
> high bit depths and 5-level transforms the coefficients would overflow,
> causing huge artifacts to appear. This was discovered thanks to the fate
> tests, which will have to be redone as this fixes a multitude of
> problems and increases PSNR.
>
> There is a slight performance drop associated with this change, making
> the encoder slower by 1.15 times, however this is necessary in order to
> avoid undefined behavior and overflows.
>
> It would be worth to template the transforms to keep the performance for
> 8 bit images as 32 bit coefficients are unnecessary for that case, but
> the primary use of the encoder is to encode video at 10 bits.
>
> Signed-off-by: Rostislav Pehlivanov <atomnuker at gmail.com>
> ---
>  libavcodec/vc2enc_dwt.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/libavcodec/vc2enc_dwt.h b/libavcodec/vc2enc_dwt.h
> index af6fe3e..7fbbfbe 100644
> --- a/libavcodec/vc2enc_dwt.h
> +++ b/libavcodec/vc2enc_dwt.h
> @@ -25,7 +25,7 @@
>  #include <stddef.h>
>  #include <stdint.h>
>
> -typedef int16_t dwtcoef;
> +typedef int32_t dwtcoef;
>
>  enum VC2TransformType {
>      VC2_TRANSFORM_9_7    = 0,   /* Deslauriers-Dubuc (9,7)  */
> --
> 2.8.1.369.geae769a
>
>

Pushed
Thanks for the reviews