[FFmpeg-devel] [PATCH 07/17] avcodec/pcm-blurayenc: Don't presume every channel layout to be native

[Andreas Rheinhardt]

The pcm_bluray encoder has AVCodec.ch_layouts set, so that
ff_encode_preinit() checks that the channel layout in use
is equivalent to one of the layouts from AVCodec.ch_layouts.
Yet equivalent is not the same as identical; in particular,
custom layouts equivalent to native layouts are possible
(and necessary if one wants to use the name/opaque fields
with an ordinary channel layout), so one must not simply
use AVChannelLayout.u.mask. Use av_channel_layout_subset()
instead.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at outlook.com>
---
 libavcodec/pcm-blurayenc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavcodec/pcm-blurayenc.c b/libavcodec/pcm-blurayenc.c
index 1f1a3a27d8..03ed88b8ae 100644
--- a/libavcodec/pcm-blurayenc.c
+++ b/libavcodec/pcm-blurayenc.c
@@ -63,7 +63,7 @@ static av_cold int pcm_bluray_encode_init(AVCodecContext *avctx)
         return AVERROR_BUG;
     }
 
-    switch (avctx->ch_layout.u.mask) {
+    switch (av_channel_layout_subset(&avctx->ch_layout, ~(uint64_t)0)) {
     case AV_CH_LAYOUT_MONO:
         ch_layout = 1;
         break;
-- 
2.34.1