[FFmpeg-cvslog] avcodec/videotoolboxenc: Add AYUV as a candidate pix_fmt for HEVC alpha

[Zhao Zhili]

ffmpeg | branch: master | Zhao Zhili <zhilizhao at tencent.com> | Tue Mar 11 21:46:58 2025 +0800| [94beaf48723adb01d452fddaa99f384350ed5c41] | committer: Zhao Zhili

avcodec/videotoolboxenc: Add AYUV as a candidate pix_fmt for HEVC alpha

Signed-off-by: Zhao Zhili <zhilizhao at tencent.com>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=94beaf48723adb01d452fddaa99f384350ed5c41
---

 libavcodec/videotoolboxenc.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/libavcodec/videotoolboxenc.c b/libavcodec/videotoolboxenc.c
index ff8f01b2ce..e7f2c7b8eb 100644
--- a/libavcodec/videotoolboxenc.c
+++ b/libavcodec/videotoolboxenc.c
@@ -1342,8 +1342,10 @@ static int vtenc_create_encoder(AVCodecContext   *avctx,
         }
     }
 
-    if (vtctx->codec_id == AV_CODEC_ID_HEVC) {
-        if (avctx->pix_fmt == AV_PIX_FMT_BGRA && vtctx->alpha_quality > 0.0) {
+    if (vtctx->codec_id == AV_CODEC_ID_HEVC && vtctx->alpha_quality > 0.0) {
+        const AVPixFmtDescriptor *descriptor = av_pix_fmt_desc_get(avctx->pix_fmt);
+
+        if (descriptor->flags & AV_PIX_FMT_FLAG_ALPHA) {
             CFNumberRef alpha_quality_num = CFNumberCreate(kCFAllocatorDefault,
                                                            kCFNumberDoubleType,
                                                            &vtctx->alpha_quality);
@@ -2860,6 +2862,7 @@ static const enum AVPixelFormat hevc_pix_fmts[] = {
     AV_PIX_FMT_NV12,
     AV_PIX_FMT_YUV420P,
     AV_PIX_FMT_BGRA,
+    AV_PIX_FMT_AYUV,
     AV_PIX_FMT_P010LE,
     AV_PIX_FMT_P210,
     AV_PIX_FMT_NONE