[FFmpeg-devel] [PATCH 1/2] avcodec/libx264: leave full range flag unchanged if unknown

[Jan Ekström]

By default the x264 full range flag is set to -1. By not setting
it to something else, we can let libx264 handle the RGB case.
Additionally, change the preference order to user-specified range
first, and then any fall-back logic left for the YUVJ pix_fmts.

Fixes the capture part of #9374
---
 libavcodec/libx264.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/libavcodec/libx264.c b/libavcodec/libx264.c
index 9afaf19547..d48e142e41 100644
--- a/libavcodec/libx264.c
+++ b/libavcodec/libx264.c
@@ -857,10 +857,12 @@ static av_cold int X264_init(AVCodecContext *avctx)
 
     x4->params.i_slice_count  = avctx->slices;
 
-    x4->params.vui.b_fullrange = avctx->pix_fmt == AV_PIX_FMT_YUVJ420P ||
-                                 avctx->pix_fmt == AV_PIX_FMT_YUVJ422P ||
-                                 avctx->pix_fmt == AV_PIX_FMT_YUVJ444P ||
-                                 avctx->color_range == AVCOL_RANGE_JPEG;
+    if (avctx->color_range != AVCOL_RANGE_UNSPECIFIED)
+        x4->params.vui.b_fullrange = avctx->color_range == AVCOL_RANGE_JPEG;
+    else if (avctx->pix_fmt == AV_PIX_FMT_YUVJ420P ||
+             avctx->pix_fmt == AV_PIX_FMT_YUVJ422P ||
+             avctx->pix_fmt == AV_PIX_FMT_YUVJ444P)
+        x4->params.vui.b_fullrange = 1;
 
     if (avctx->colorspace != AVCOL_SPC_UNSPECIFIED)
         x4->params.vui.i_colmatrix = avctx->colorspace;
-- 
2.31.1