[FFmpeg-devel] [PATCH] avcodec/vp8: Fix wrong vpx_rac_is_end() check

[Ronald S. Bultje]

Hi,

On Wed, Nov 16, 2022 at 8:20 AM Ronald S. Bultje <rsbultje at gmail.com> wrote:

> On Wed, Nov 16, 2022 at 3:44 AM Hirokazu Honda <hiroh at chromium.org> wrote:
>
>> @@ -2471,7 +2472,7 @@ static av_always_inline int
>> decode_mb_row_no_filter(AVCodecContext *avctx, void
>>          prefetch_motion(s, mb, mb_x, mb_y, mb_xy, VP8_FRAME_PREVIOUS);
>>
>>          if (!mb->skip)
>> -            decode_mb_coeffs(s, td, c, mb, s->top_nnz[mb_x],
>> td->left_nnz, is_vp7);
>> +            decode_mb_coeffs(s, td, coeff_c, mb, s->top_nnz[mb_x],
>> td->left_nnz, is_vp7);
>>
>>          if (mb->mode <= MODE_I4x4)
>>              intra_predict(s, td, dst, mb, mb_x, mb_y, is_vp7);
>> --
>> 2.38.1.431.g37b22c650d-goog
>>
>
> Thanks for the fix. So, you're transferring the truncation check from the
> per-slice coef buffer to the "main" mode/mv buffer. Shouldn't we check
> both? Or is one of them allowed to be truncated? (That seems unlikely, but
> that appears to be what you're suggesting here, otherwise there wouldn't be
> any artifacts.)
>

To clarify: I think a vpx_rac_is_end(coeff_c) in the "if (!mb->skip)"
branch here would be helpful, I think. Outside, we don't use coeff_c, so no
truncation check is needed. Assuming that still fixes the issue, then.

Ronald