[FFmpeg-cvslog] lavc/vvc: Fix derivation of inverse LMCS idx

[Frank Plowman]

ffmpeg | branch: master | Frank Plowman <post at frankplowman.com> | Sun Feb  2 12:10:48 2025 +0000| [392aa9daf2c730465f6aba5ba2b37e02695de9f7] | committer: Nuo Mi

lavc/vvc: Fix derivation of inverse LMCS idx

The clamping of idxYInv from H.266(V3) section 8.8.2.3 was missing.
This could lead to OOB reads from lmcs->pivot or input_pivot.

I also changed the derivation of the forward LMCS idx to use a shift
rather than a division for speed and as this is actually how the
variable is declared in the specification (8.7.5.2).

Signed-off-by: Frank Plowman <post at frankplowman.com>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=392aa9daf2c730465f6aba5ba2b37e02695de9f7
---

 libavcodec/vvc/ps.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/libavcodec/vvc/ps.c b/libavcodec/vvc/ps.c
index d026e797b3..5ae423a737 100644
--- a/libavcodec/vvc/ps.c
+++ b/libavcodec/vvc/ps.c
@@ -835,7 +835,7 @@ static int lmcs_derive_lut(VVCLMCS *lmcs, const H266RawAPS *rlmcs, const H266Raw
 
     //derive lmcs_fwd_lut
     for (uint16_t sample = 0; sample < max; sample++) {
-        const int idx_y = sample / org_cw;
+        const int idx_y = sample >> shift;
         const uint16_t fwd_sample = lmcs_derive_lut_sample(sample, lmcs->pivot,
             input_pivot, scale_coeff, idx_y, max);
         if (bit_depth > 8)
@@ -851,6 +851,7 @@ static int lmcs_derive_lut(VVCLMCS *lmcs, const H266RawAPS *rlmcs, const H266Raw
         uint16_t inv_sample;
         while (i <= lmcs->max_bin_idx && sample >= lmcs->pivot[i + 1])
             i++;
+        i = FFMIN(i, LMCS_MAX_BIN_SIZE - 1);
 
         inv_sample = lmcs_derive_lut_sample(sample, input_pivot, lmcs->pivot,
             inv_scale_coeff, i, max);