[FFmpeg-devel] [PATCH] avcodec/ppc/vp8dsp_altivec: Fix out-of-bounds access

[Sean McGovern]

Andreas:

On Tue, Mar 12, 2024 at 9:14 PM Andreas Rheinhardt
<andreas.rheinhardt at outlook.com> wrote:
>
> h_subpel_filters_inner[i] and h_subpel_filters_outer[i / 2]
> belong together and the former allows the range 0..6,
> so the latter needs to support 0..3. But it has only three
> elements. Add another one.
> The value for the last element has been guesstimated
> from subpel_filters in libavcodec/vp8dsp.c.
>
> This is also intended to fix FATE-failures with UBSan here:
> https://fate.ffmpeg.org/report.cgi?time=20240312011016&slot=ppc-linux-gcc-13.2-ubsan-altivec-qemu
>
> Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at outlook.com>
> ---
>  libavcodec/ppc/vp8dsp_altivec.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/libavcodec/ppc/vp8dsp_altivec.c b/libavcodec/ppc/vp8dsp_altivec.c
> index 12dac8b0a8..061914fc38 100644
> --- a/libavcodec/ppc/vp8dsp_altivec.c
> +++ b/libavcodec/ppc/vp8dsp_altivec.c
> @@ -50,11 +50,12 @@ static const vec_s8 h_subpel_filters_inner[7] =
>  // for 6tap filters, these are the outer two taps
>  // The zeros mask off pixels 4-7 when filtering 0-3
>  // and vice-versa
> -static const vec_s8 h_subpel_filters_outer[3] =
> +static const vec_s8 h_subpel_filters_outer[4] =
>  {
>      REPT4(0, 0, 2, 1),
>      REPT4(0, 0, 3, 3),
>      REPT4(0, 0, 1, 2),
> +    REPT4(0, 0, 0, 0),
>  };
>
>  #define LOAD_H_SUBPEL_FILTER(i) \
> --
> 2.40.1
>
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".

Confirming this patch fixes fate-checkasm-vp8dsp (and presumably the
other vp8 tests in GCC UBsan) on PowerPC QEMU, POWER7 (ppc64), and
POWER9 (ppc64le).

Thanks again,
Sean McGovern