[FFmpeg-devel] [PATCH 11/15] avfilter/palettegen: comment on the unnormalized variance

[Clément Bœsch]

---
 libavfilter/vf_palettegen.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/libavfilter/vf_palettegen.c b/libavfilter/vf_palettegen.c
index 2b412cdb55..b8e4463539 100644
--- a/libavfilter/vf_palettegen.c
+++ b/libavfilter/vf_palettegen.c
@@ -152,6 +152,24 @@ static void compute_box_variance(PaletteGenContext *s, struct range_box *box)
         const struct color_ref *ref = s->refs[box->start + i];
         variance += diff(ref->color, box->color) * ref->count;
     }
+    /*
+     * The variance is computed as a Mean Squared Error of the distance of the
+     * current color to the box color average, with an important difference:
+     * the final sum is not normalized using the total weight of the box (the
+     * sum of the ref->count).
+     *
+     * One may expect that in order to compare the variance of the boxes
+     * between each others a normalization makes sense. Unfortunately, the
+     * normalization has the side effect of taking the "size" of the box out of
+     * the equation. In practice, this has a tendency to cause dramatic banding
+     * effects (in particular without dithering). Typically a scene where the
+     * sky is omnipresent may get much less colors assigned than its
+     * surroundings.
+     *
+     * Not normalizing causes a bias towards boxes with many colors, but that's
+     * exactly what we want. The downside is that vivid/accent colors found in
+     * small quantities tend to disappear.
+     */
     box->variance = variance;
 }
 
-- 
2.38.1