[FFmpeg-devel] [PATCH] avcodec/vc2enc_dwt: Fix left shifts of negative numbers

[Andreas Rheinhardt]

Andreas Rheinhardt:
> Fixes the vsynth(1|2|_lena)-vc2-(thaar|t5_3) FATE tests.
> 
> Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at outlook.com>
> ---
>  libavcodec/vc2enc_dwt.c | 7 +++----
>  1 file changed, 3 insertions(+), 4 deletions(-)
> 
> diff --git a/libavcodec/vc2enc_dwt.c b/libavcodec/vc2enc_dwt.c
> index 441af040ec..809f938e1b 100644
> --- a/libavcodec/vc2enc_dwt.c
> +++ b/libavcodec/vc2enc_dwt.c
> @@ -151,7 +151,7 @@ static void vc2_subband_dwt_53(VC2TransformContext *t, dwtcoef *data,
>       */
>      for (y = 0; y < synth_height; y++) {
>          for (x = 0; x < synth_width; x++)
> -            synthl[x] = datal[x] << 1;
> +            synthl[x] = datal[x] * 2;
>          synthl += synth_width;
>          datal  += stride;
>      }
> @@ -223,9 +223,8 @@ static av_always_inline void dwt_haar(VC2TransformContext *t, dwtcoef *data,
>      /* Horizontal synthesis. */
>      for (y = 0; y < synth_height; y++) {
>          for (x = 0; x < synth_width; x += 2) {
> -            synthl[y*synth_width + x + 1] = (datal[y*stride + x + 1] << s) -
> -                                            (datal[y*stride + x] << s);
> -            synthl[y*synth_width + x] = (datal[y*stride + x + 0] << s) +
> +            synthl[y*synth_width + x + 1] = (datal[y*stride + x + 1] - datal[y*stride + x]) * (1 << s);
> +            synthl[y*synth_width + x] = datal[y*stride + x + 0] * (1 << s) +
>                                          ((synthl[y*synth_width + x + 1] + 1) >> 1);
>          }
>      }

Will apply this patch tomorrow unless there are objections.

- Andreas