[FFmpeg-devel] [PATCH v3] lavc/aomdec: Allow RGB decoding

[Valerii Zapodovnikov]

Yes, RGB is signalled by Identity matrix if and only if XYZ is not
in transfer. XYZ primaries are just normal primaries that can be
used for normal RGB, no problem, so I do not check for them.
No need to test for sRGB primaries (that is AVCOL_PRI_BT709), as
ffplay does not know what that is (is not color managed), but mpv
will do that automatically. This will also support other transfers
like DCI-P3 / DCI-D65 one, et cetera. See libvpxdec.c.
Also the default AV1 decoder is libdav1d, which is not affected.
For XYZ support someone should add correct pixel format in the code.

Signed-off-by: Valerii Zapodovnikov <val.zapod.vz at gmail.com>
---
 libavcodec/libaomdec.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)

diff --git a/libavcodec/libaomdec.c b/libavcodec/libaomdec.c
index 6e7324a832..156e644263 100644
--- a/libavcodec/libaomdec.c
+++ b/libavcodec/libaomdec.c
@@ -134,15 +134,27 @@ static int set_pix_fmt(AVCodecContext *avctx, struct aom_image *img)
     case AOM_IMG_FMT_I444:
     case AOM_IMG_FMT_I44416:
         if (img->bit_depth == 8) {
-            avctx->pix_fmt = AV_PIX_FMT_YUV444P;
+            if (avctx->colorspace == AVCOL_SPC_RGB && avctx->color_trc != AVCOL_TRC_SMPTE428) {
+                avctx->pix_fmt = AV_PIX_FMT_GBRP;
+            } else {
+                avctx->pix_fmt = AV_PIX_FMT_YUV444P;
+            }
             avctx->profile = FF_PROFILE_AV1_HIGH;
             return 0;
         } else if (img->bit_depth == 10) {
-            avctx->pix_fmt = AV_PIX_FMT_YUV444P10;
+            if (avctx->colorspace == AVCOL_SPC_RGB && avctx->color_trc != AVCOL_TRC_SMPTE428) {
+                avctx->pix_fmt = AV_PIX_FMT_GBRP10;
+            } else {
+                avctx->pix_fmt = AV_PIX_FMT_YUV444P10;
+            }
             avctx->profile = FF_PROFILE_AV1_HIGH;
             return 0;
         } else if (img->bit_depth == 12) {
-            avctx->pix_fmt = AV_PIX_FMT_YUV444P12;
+            if (avctx->colorspace == AVCOL_SPC_RGB && avctx->color_trc != AVCOL_TRC_SMPTE428) {
+                avctx->pix_fmt = AV_PIX_FMT_GBRP12;
+            } else {
+                avctx->pix_fmt = AV_PIX_FMT_YUV444P12;
+            }
             avctx->profile = FF_PROFILE_AV1_PROFESSIONAL;
             return 0;
         } else {
-- 
2.30.2