[FFmpeg-devel] [PATCH 4/4] avcodec/svq1enc: Use unsigned for parameter >= 0 to workaround GCC bug

[Andreas Rheinhardt]

encode_block() in svq1enc.c looks like the following:

static int encode_block(int block[7][256], int level)
{
    int best_score = 0;

    for (unsigned x = 0; x < level; x++) {
        int v = block[1][x];
        block[level][x] = 0;
        best_score      += v * v;
    }

    if (level > 0 && best_score > 64) {
        int score = 0;

        score += encode_block(block, level - 1);
        score += encode_block(block, level - 1);

        if (score < best_score) {
            best_score = score;
        }
    }

    return best_score;
}

When called from outside of encode_block(), it is always called with
level == 5.

This triggers a bug [1] in GCC: On -O3, it creates eight clones of
encode_block with different values of level inlined into it. The clones
with negative values are of course useless*, but they also lead to
-Warray-bounds warnings, because they access block[-1].

This has been mitigated in GCC 12: It no longer creates clones
for parameters that it knows are impossible. Somehow switching levels
to unsigned makes GCC know this. Therefore this commit does this.
(For GCC 11, this changes the warning to "array subscript 4294967295 is
above array bounds" from "array subscript -1 is below array bounds".)

[1]: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=102513

*: These clones can actually be discarded when compiling with
-ffunction-sections.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at outlook.com>
---
 libavcodec/svq1enc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavcodec/svq1enc.c b/libavcodec/svq1enc.c
index 3c2d594632..6072f8d07d 100644
--- a/libavcodec/svq1enc.c
+++ b/libavcodec/svq1enc.c
@@ -91,7 +91,7 @@ static int ssd_int8_vs_int16_c(const int8_t *pix1, const int16_t *pix2,
 }
 
 static int encode_block(SVQ1EncContext *s, uint8_t *src, uint8_t *ref,
-                        uint8_t *decoded, int stride, int level,
+                        uint8_t *decoded, int stride, unsigned level,
                         int threshold, int lambda, int intra)
 {
     int count, y, x, i, j, split, best_mean, best_score, best_count;
-- 
2.34.1