[FFmpeg-devel] [PATCH] avcodec/aacdec_fixed: Fix integer overflow

[Michael Niedermayer]

Hi

On Wed, Jul 29, 2015 at 11:58:43AM +0200, Nedeljko Babic wrote:
> Add type cast of result of av_clipl_int32() to 64 bit to avoid overflow in
> addition later.
> 
> Fixes fate failure with clang ftrapv.
> 
> Signed-off-by: Nedeljko Babic <nedeljko.babic at imgtec.com>
> ---
>  libavcodec/aacdec_template.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/libavcodec/aacdec_template.c b/libavcodec/aacdec_template.c
> index 13653a8..eb14bd9 100644
> --- a/libavcodec/aacdec_template.c
> +++ b/libavcodec/aacdec_template.c
> @@ -2749,8 +2749,8 @@ static void spectral_to_sample(AACContext *ac, int samples)
>                      int j;
>                      /* preparation for resampler */
>                      for(j = 0; j<samples; j++){
> -                        che->ch[0].ret[j] = (int32_t)av_clipl_int32((int64_t)che->ch[0].ret[j]<<7)+0x8000;
> -                        che->ch[1].ret[j] = (int32_t)av_clipl_int32((int64_t)che->ch[1].ret[j]<<7)+0x8000;
> +                        che->ch[0].ret[j] = (int32_t)((int64_t)av_clipl_int32((int64_t)che->ch[0].ret[j]<<7)+0x8000);
> +                        che->ch[1].ret[j] = (int32_t)((int64_t)av_clipl_int32((int64_t)che->ch[1].ret[j]<<7)+0x8000);

why does it overflow ?
also all overflows during fate occur in the 2nd channel only
are the affected case maybe all just using the first channel, or
something like that ?
it seems enough to skip the 2nd depending on type

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Avoid a single point of failure, be that a person or equipment.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20150729/b823337a/attachment.sig>