[FFmpeg-devel] [PATCH] avcodec/vvc/ctu: check coeff before multiply (PR #20142)

[Kacper Michajłow]

PR #20142 opened by Kacper Michajłow (kasper93)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20142
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20142.patch

ff_vvc_palette_escape_val() can return AVERROR in which case the
coeff*scale will overflow.

Fixes: runtime error: signed integer overflow: -1094995529 * 6528 cannot
       be represented in type 'int'
Fixes: OSS-Fuzz/435225406


From aa5df295b5e5958c30ff07db482d58eba6009b25 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Kacper=20Michaj=C5=82ow?= <kasper93 at gmail.com>
Date: Wed, 6 Aug 2025 18:58:10 +0200
Subject: [PATCH] avcodec/vvc/ctu: check coeff before multiply
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

ff_vvc_palette_escape_val() can return AVERROR in which case the
coeff*scale will overflow.

Fixes: runtime error: signed integer overflow: -1094995529 * 6528 cannot
       be represented in type 'int'
Fixes: OSS-Fuzz/435225406
Signed-off-by: Kacper Michajłow <kasper93 at gmail.com>
---
 libavcodec/vvc/ctu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavcodec/vvc/ctu.c b/libavcodec/vvc/ctu.c
index d54e6a322b..fd7d07f438 100644
--- a/libavcodec/vvc/ctu.c
+++ b/libavcodec/vvc/ctu.c
@@ -2054,9 +2054,9 @@ static int palette_subblock_data(VVCLocalContext *lc,
                 const int v = PALETTE_INDEX(xc, yc);
                 if (v == esc) {
                     const int coeff = ff_vvc_palette_escape_val(lc, (1 << sps->bit_depth) - 1);
-                    const int pixel = av_clip_intp2(RSHIFT(coeff * scale, 6), sps->bit_depth);
                     if (coeff < 0)
                         return AVERROR_INVALIDDATA;
+                    const int pixel = av_clip_intp2(RSHIFT(coeff * scale, 6), sps->bit_depth);
                     PALETTE_SET_PIXEL(xc, yc, pixel);
                 } else {
                     PALETTE_SET_PIXEL(xc, yc, plt->entries[v]);
-- 
2.49.1