[FFmpeg-devel] [PATCH] avcodec/mlp*: improvements

[Tomas Härdin]

On Wed, 2023-10-25 at 21:00 +0200, Paul B Mahol wrote:
> On Wed, Oct 25, 2023 at 8:39 PM Tomas Härdin <git at haerdin.se> wrote:
> 
> > 
> > >             if (c) {
> > >                 e[0] = 1 << 14;
> > >                 e[1] = 0 << 14;
> > >                 e[2] = v[1];
> > >                 e[3] = v[0];
> > >             } else {
> > >                 e[0] = v[0];
> > >                 e[1] = v[1];
> > >                 e[2] = 0 << 14;
> > >                 e[3] = 1 << 14;
> > >             }
> > > 
> > >             if (invert2x2(e, d)) {
> > >                 sum = UINT64_MAX;
> > >                 goto next;
> > >             }
> > > 
> > 
> > You can make use of the properties of e to simplify calculating the
> > inverse. The determinant is always v[0]<<14, so you can just do if
> > (!v[0]) continue; and skip the determinant check altogether.
> > 
> 
> Even for real 2x2 matrix case? (Once one of rows is not 1, 0) ?
> May added such cases later.

You can just work the math out on paper. Inverse of

 1     0
 v[1]  v[0]

is

 1           0
 -v[1]/v[0]  1/v[0]

not accounting for shifts.

Also RE: my other comments, you are right. I didn't take into account
that MLP is lossless and that there may be off-by-one errors.

And as I said on IRC you can formulate this as a least squares problem,
then solve it using a linear system solve. This patch seems finds a
solution that minimizes L1 rather than L2 though. Not sure what the
implications of that are compressionwise. What happens if you replace
FFABS() with a square for scoring?

/Tomas