[FFmpeg-devel] [PATCH 4/4] avformat/vividas: Fixes overflow in shift in recover_key()

[Reimar Döffinger]

On 28.06.2019, at 22:53, Michael Niedermayer <michael at niedermayer.cc> wrote:

> Fixes: left shift of 133 by 24 places cannot be represented in type 'int'
> Fixes: 15365/clusterfuzz-testcase-minimized-ffmpeg_DEMUXER_fuzzer-5716153105645568
> 
> Found-by: continuous fuzzing process https://github.com/google/oss-fuzz/tree/master/projects/ffmpeg
> Signed-off-by: Michael Niedermayer <michael at niedermayer.cc>
> ---
> libavformat/vividas.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/libavformat/vividas.c b/libavformat/vividas.c
> index 753328245d..ed2eaea633 100644
> --- a/libavformat/vividas.c
> +++ b/libavformat/vividas.c
> @@ -118,7 +118,7 @@ static unsigned recover_key(unsigned char sample[4], unsigned expected_size)
>     return (sample[0]^plaintext[0])|
>         ((sample[1]^plaintext[1])<<8)|
>         ((sample[2]^plaintext[2])<<16)|
> -        ((sample[3]^plaintext[3])<<24);
> +        ((unsigned)(sample[3]^plaintext[3])<<24);

Shouldn't this just be
return AV_RL32(sample) ^ AV_RL32(plaintext);
?
If so, the code might be worthy of review for more needless overcomplication...