[FFmpeg-devel] [PATCH] avcodec/nvdec_mpeg2: fix order of quant matrix coefficients

[Hendrik Leppkes]

On Mon, May 15, 2023 at 12:51 PM Hendrik Leppkes <h.leppkes at gmail.com> wrote:
>
> mpeg2dec stores them permutated for the IDCT, nvdec expects them in
> plain raster order.
> ---
>  libavcodec/nvdec_mpeg12.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/libavcodec/nvdec_mpeg12.c b/libavcodec/nvdec_mpeg12.c
> index e10735587d..3b9ff60734 100644
> --- a/libavcodec/nvdec_mpeg12.c
> +++ b/libavcodec/nvdec_mpeg12.c
> @@ -83,8 +83,9 @@ static int nvdec_mpeg12_start_frame(AVCodecContext *avctx, const uint8_t *buffer
>      };
>
>      for (i = 0; i < 64; ++i) {
> -        ppc->QuantMatrixIntra[i] = s->intra_matrix[i];
> -        ppc->QuantMatrixInter[i] = s->inter_matrix[i];
> +        int n = s->idsp.idct_permutation[i];
> +        ppc->QuantMatrixIntra[i] = s->intra_matrix[n];
> +        ppc->QuantMatrixInter[i] = s->inter_matrix[n];
>      }
>
>      return 0;
> --
> 2.40.1.windows.1
>

Apparently mpeg4 and VDPAU have the same issue. I can test and fix
mpeg4, but do not have VDPAU setup, so .. untested commits incoming?

- Hendrik