[FFmpeg-devel] [PATCH] Revert "avfilter/vf_palette(gen|use): support palettes with alpha"

[Soft Works]

> -----Original Message-----
> From: ffmpeg-devel <ffmpeg-devel-bounces at ffmpeg.org> On Behalf Of
> Clément Bœsch
> Sent: Monday, October 31, 2022 1:30 AM
> To: FFmpeg development discussions and patches <ffmpeg-
> devel at ffmpeg.org>
> Subject: Re: [FFmpeg-devel] [PATCH] Revert
> "avfilter/vf_palette(gen|use): support palettes with alpha"
> 
> On Sun, Oct 30, 2022 at 10:55:31PM +0000, Soft Works wrote:
> [...]
> 
> > Do you think it might make sense to put more weight on the
> > alpha value by tripling it? So it would be weighted equally to the
> > RGB value?
> 
> You cannot mix alpha with colors at all, they are separate domains
> and you
> need to treat them as such.
> 
> From paletteuse perspective what you need to do is first choose the
> colors
> in the palette that match exactly the alpha (or at least the closest
> if
> and only there is no exact match). Then within that set, and only
> within
> that one, you'd pick the closest color.
> 
> From palettegen perspective, you need to split the colors in
> different
> transparency domain (a first dimensional quantization), then quantize
> the
> colors in each quantized alpha dimension. And when you have all your
> quantized palettes for each level of alpha, you find an algorithm to
> reduce the number of transparency dimensions or the number of colors
> per
> dimension to make it fit inside a single palette. But you can't just
> do
> the alpha and the colors at the same time, it cannot work, whatever
> weights you choose.

Interestingly, pngquant which is supposed to have the best open source
quantization algorithms seems to be using weights (albeit in a more 
sophisticated way) and does not handle alpha separately for calculating 
color distance, variance and averaging:

https://github.com/ImageOptim/libimagequant/blob/a16c9ca66a24158496da02d86925cc0167831205/pam.h#L163-L182
https://github.com/ImageOptim/libimagequant/blob/a16c9ca66a24158496da02d86925cc0167831205/mediancut.c#L29-L49
https://github.com/ImageOptim/libimagequant/blob/a16c9ca66a24158496da02d86925cc0167831205/mediancut.c#L449-L476


softworkz