[FFmpeg-devel] [PATCH] fixed luma quantizing when q >= MAX_STORED_Q

[Alex Mogurenko]

problem occurs in slice quant estimation and slice encoding.

if slice quant >= MAX_STORED_Q we dont use pre-calculated quant matrices
but generate new:

qmat = ctx->custom_q;

        qmat_chroma = ctx->custom_q;

        for (i = 0; i < 64; i++) {

            qmat[i] = ctx->quant_mat[i] * quant;

            qmat_chroma[i] = ctx->quant_chroma_mat[i] * quant;

        }


as you see both qmat and qmat_chroma both point to same ctx -> custom_q

as result they will contain chroma qunatizers as


qmat_chroma[i] = ctx->quant_chroma_mat[i] * quant;


last in loop, after all we pass qmat/qmat_chroma to function where we
estimate encoded slice size or encode slice:

estimate_slice_plane / encode_slice_plane


I will make new patch with detailed description


пт, 28 дек. 2018 г. в 18:43, Derek Buitenhuis <derek.buitenhuis at gmail.com>:

> On 27/12/2018 19:28, Alex Mogurenko wrote:
> > ---
> >  libavcodec/proresenc_kostya.c | 6 ++++--
> >  1 file changed, 4 insertions(+), 2 deletions(-)
>
> Can you give a little more detail about what's changed and why,
> in the commit message?
>
> It looks like custom_chroma_q is zero initialized and never set to
> anything?
>
> - Derek
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> http://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>