[FFmpeg-cvslog] swscale/swscale_unscaled: fix packed16togbra16() for formats with bpc between 9-14 bits

[Ramiro Polla]

ffmpeg | branch: master | Ramiro Polla <ramiro.polla at gmail.com> | Sun May 18 22:52:09 2025 +0200| [748e960e04248569d121ac4ebba11109b6c81acb] | committer: Ramiro Polla

swscale/swscale_unscaled: fix packed16togbra16() for formats with bpc between 9-14 bits

Currently, packed16togbra16() always sets the alpha value to 0xFFFF,
without taking the bit depth into consideration.

This causes a bug on x86, which can be reproduced with:
./libswscale/tests/swscale -unscaled 1 -src xyz12le -dst gbrap12be

The problem arises in ff_hscale14to15_4_ssse3(), in the conversion
from gbrap12be to yuva444p, which comes after the conversion from
xyz12le to gbrap12be.

It has something to do with pmaddwd not working on unsigned values.
There is some code to deal with 0xFFFF if the input has a bit depth of
16, but not for bit depths < 16.
We could fix ff_hscale14to15_4_ssse3() to also work correctly with
0xFFFF on bit depths < 16, or we could just not write 0xFFFF there in
the first place, which is what this commit does.

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=748e960e04248569d121ac4ebba11109b6c81acb
---

 libswscale/swscale_unscaled.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/libswscale/swscale_unscaled.c b/libswscale/swscale_unscaled.c
index 8d71a88c23..488db43e0b 100644
--- a/libswscale/swscale_unscaled.c
+++ b/libswscale/swscale_unscaled.c
@@ -699,7 +699,7 @@ static void packed16togbra16(const uint8_t *src, int srcStride,
                     dst[0][x] = av_bswap16(av_bswap16(*src_line++) >> shift);
                     dst[1][x] = av_bswap16(av_bswap16(*src_line++) >> shift);
                     dst[2][x] = av_bswap16(av_bswap16(*src_line++) >> shift);
-                    dst[3][x] = 0xFFFF;
+                    dst[3][x] = av_bswap16(0xFFFF >> shift);
                 }
             } else if (src_alpha) {
                 for (x = 0; x < width; x++) {
@@ -729,7 +729,7 @@ static void packed16togbra16(const uint8_t *src, int srcStride,
                     dst[0][x] = av_bswap16(*src_line++ >> shift);
                     dst[1][x] = av_bswap16(*src_line++ >> shift);
                     dst[2][x] = av_bswap16(*src_line++ >> shift);
-                    dst[3][x] = 0xFFFF;
+                    dst[3][x] = av_bswap16(0xFFFF >> shift);
                 }
             } else if (src_alpha) {
                 for (x = 0; x < width; x++) {
@@ -759,7 +759,7 @@ static void packed16togbra16(const uint8_t *src, int srcStride,
                     dst[0][x] = av_bswap16(*src_line++) >> shift;
                     dst[1][x] = av_bswap16(*src_line++) >> shift;
                     dst[2][x] = av_bswap16(*src_line++) >> shift;
-                    dst[3][x] = 0xFFFF;
+                    dst[3][x] = 0xFFFF >> shift;
                 }
             } else if (src_alpha) {
                 for (x = 0; x < width; x++) {
@@ -789,7 +789,7 @@ static void packed16togbra16(const uint8_t *src, int srcStride,
                     dst[0][x] = *src_line++ >> shift;
                     dst[1][x] = *src_line++ >> shift;
                     dst[2][x] = *src_line++ >> shift;
-                    dst[3][x] = 0xFFFF;
+                    dst[3][x] = 0xFFFF >> shift;
                 }
             } else if (src_alpha) {
                 for (x = 0; x < width; x++) {