[FFmpeg-cvslog] vp9: recon: Use emulated edge to prevent buffer overflows

[Janne Grunau]

ffmpeg | branch: release/5.1 | Janne Grunau <janne-ffmpeg at jannau.net> | Fri Jan  3 01:54:40 2025 +0100| [14afc43c27d91f390389ad69b0b8d4056b349a64] | committer: Ronald S. Bultje

vp9: recon: Use emulated edge to prevent buffer overflows

The arm/aarch64 horizontal filter reads one additional pixel beyond what
the filter uses. This can become an issue if the application does not
allocate larger buffers than what's required for the pixel data. If the
motion vector points to the bottom right edge of the picture this
becomes a read buffer overflow. This triggers segfaults in Firefox for
video resolutions which result in a page aligned picture size like
1280x640.
Prevent this by using emulated edge in this case.

Fixes: https://bugzilla.mozilla.org/show_bug.cgi?id=1881185
Signed-off-by: Janne Grunau <janne-ffmpeg at jannau.net>
Signed-off-by: Ronald S. Bultje <rsbultje at gmail.com>
(cherry picked from commit 060464105bdca82b8cfb91c7a6fb56df4c7cd9d0)

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=14afc43c27d91f390389ad69b0b8d4056b349a64
---

 libavcodec/vp9recon.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/libavcodec/vp9recon.c b/libavcodec/vp9recon.c
index bfafde8c9c..ce04244ccb 100644
--- a/libavcodec/vp9recon.c
+++ b/libavcodec/vp9recon.c
@@ -318,7 +318,11 @@ static av_always_inline void mc_luma_unscaled(VP9TileData *td, vp9_mc_func (*mc)
     // The arm/aarch64 _hv filters read one more row than what actually is
     // needed, so switch to emulated edge one pixel sooner vertically
     // (!!my * 5) than horizontally (!!mx * 4).
+    // The arm/aarch64 _h filters read one more pixel than what actually is
+    // needed, so switch to emulated edge if that would read beyond the bottom
+    // right block.
     if (x < !!mx * 3 || y < !!my * 3 ||
+        ((ARCH_AARCH64 || ARCH_ARM) && (x + !!mx * 5 > w - bw) && (y + !!my * 5 + 1 > h - bh)) ||
         x + !!mx * 4 > w - bw || y + !!my * 5 > h - bh) {
         s->vdsp.emulated_edge_mc(td->edge_emu_buffer,
                                  ref - !!my * 3 * ref_stride - !!mx * 3 * bytesperpixel,
@@ -357,7 +361,11 @@ static av_always_inline void mc_chroma_unscaled(VP9TileData *td, vp9_mc_func (*m
     // The arm/aarch64 _hv filters read one more row than what actually is
     // needed, so switch to emulated edge one pixel sooner vertically
     // (!!my * 5) than horizontally (!!mx * 4).
+    // The arm/aarch64 _h filters read one more pixel than what actually is
+    // needed, so switch to emulated edge if that would read beyond the bottom
+    // right block.
     if (x < !!mx * 3 || y < !!my * 3 ||
+        ((ARCH_AARCH64 || ARCH_ARM) && (x + !!mx * 5 > w - bw) && (y + !!my * 5 + 1 > h - bh)) ||
         x + !!mx * 4 > w - bw || y + !!my * 5 > h - bh) {
         s->vdsp.emulated_edge_mc(td->edge_emu_buffer,
                                  ref_u - !!my * 3 * src_stride_u - !!mx * 3 * bytesperpixel,