[FFmpeg-devel] [PATCH] checkasm/h264dsp: Fix stack overflow in check_idct_dequant

[Zhao Zhili]

> On Jun 16, 2025, at 15:16, Andreas Rheinhardt <andreas.rheinhardt at outlook.com> wrote:
> 
> Zhao Zhili:
>> From: Zhao Zhili <zhilizhao at tencent.com>
>> 
>> ---
>> tests/checkasm/h264dsp.c | 14 ++++++++++----
>> 1 file changed, 10 insertions(+), 4 deletions(-)
>> 
>> diff --git a/tests/checkasm/h264dsp.c b/tests/checkasm/h264dsp.c
>> index f5f9650224..006532e08b 100644
>> --- a/tests/checkasm/h264dsp.c
>> +++ b/tests/checkasm/h264dsp.c
>> @@ -328,7 +328,7 @@ static void check_idct_multiple(void)
>> static void check_idct_dequant(void)
>> {
>>     static const int depths[5] = { 8, 9, 10, 12, 14 };
>> -    LOCAL_ALIGNED_16(int16_t, src, [16]);
>> +    LOCAL_ALIGNED_16(int16_t, src, [16 * 2]);
>>     /* Ensure dst buffers are large enough to hold dctcoefs of all bit-depths. */
>>     LOCAL_ALIGNED_16(uint8_t, dst0, [16 * 16 * sizeof(int32_t)]);
>>     LOCAL_ALIGNED_16(uint8_t, dst1, [16 * 16 * sizeof(int32_t)]);
>> @@ -338,15 +338,21 @@ static void check_idct_dequant(void)
>>     int bit_depth, i, qmul;
>>     declare_func_emms(AV_CPU_FLAG_MMX | AV_CPU_FLAG_SSE2, void, int16_t *output, int16_t *input, int qmul);
>> 
>> -    for (int j = 0; j < 16; j++)
>> -        src[j] = (rnd() % 512) - 256;
>> -
>>     qmul = rnd() % 4096;
>> 
>>     for (i = 0; i < FF_ARRAY_ELEMS(depths); i++) {
>>         bit_depth = depths[i];
>>         ff_h264dsp_init(&h, bit_depth, 1);
>> 
>> +        if (bit_depth == 8) {
>> +            for (int j = 0; j < 16; j++)
>> +                src[j] = (rnd() % 512) - 256;
>> +        } else {
>> +            int32_t *p = (int32_t *)src;
>> +            for (int j = 0; j < 16; j++)
>> +                p[j] = (rnd() % (1 << (bit_depth + 1))) - (1 << bit_depth);
> 
> This is an effective type violation and therefore UB.

Yes. And the template functions are UB.

> Furthermore,
> increasing the size of the array has the downside that stack overflows
> in the 8 bit codepath may go undetected. So better add a
> LOCAL_ALIGNED_16(int32_t, src32, [16]) and use that for the >8 bit tests.

I think this is still UB by pass it as argument to h264_luma_dc_dequant_idct,
due to the function prototype.

I have no idea other than union or separate test case.

> 
>> +        }
>> +
>>         memset(dst0, 0, 16 * 16 * SIZEOF_COEF);
>>         memset(dst1, 0, 16 * 16 * SIZEOF_COEF);
>> 
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".