[FFmpeg-cvslog] avfilter/vf_paletteuse: do not sort transparency color

[Paul B Mahol]

ffmpeg | branch: master | Paul B Mahol <onemda at gmail.com> | Thu Aug 19 00:13:22 2021 +0200| [835eb0a556d678dd65bb1978519c4d62d93c37c4] | committer: Paul B Mahol

avfilter/vf_paletteuse: do not sort transparency color

Make last palette entry always transparent color.
Fixes wrong filtered output with new=1 option set.

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=835eb0a556d678dd65bb1978519c4d62d93c37c4
---

 libavfilter/vf_paletteuse.c | 13 ++++---------
 1 file changed, 4 insertions(+), 9 deletions(-)

diff --git a/libavfilter/vf_paletteuse.c b/libavfilter/vf_paletteuse.c
index 4e90927322..1f1e36ba8e 100644
--- a/libavfilter/vf_paletteuse.c
+++ b/libavfilter/vf_paletteuse.c
@@ -731,18 +731,13 @@ static void load_colormap(PaletteUseContext *s)
     uint32_t last_color = 0;
     struct color_rect box;
 
-    /* disable transparent colors and dups */
-    qsort(s->palette, AVPALETTE_COUNT, sizeof(*s->palette), cmp_pal_entry);
-    // update transparency index:
     if (s->transparency_index >= 0) {
-        for (i = 0; i < AVPALETTE_COUNT; i++) {
-            if ((s->palette[i]>>24 & 0xff) == 0) {
-                s->transparency_index = i; // we are assuming at most one transparent color in palette
-                break;
-            }
-        }
+        FFSWAP(uint32_t, s->palette[s->transparency_index], s->palette[255]);
     }
 
+    /* disable transparent colors and dups */
+    qsort(s->palette, AVPALETTE_COUNT-(s->transparency_index >= 0), sizeof(*s->palette), cmp_pal_entry);
+
     for (i = 0; i < AVPALETTE_COUNT; i++) {
         const uint32_t c = s->palette[i];
         if (i != 0 && c == last_color) {