[FFmpeg-cvslog] avcodec/pcm-blurayenc: Don't presume every channel layout to be native

[Andreas Rheinhardt]

ffmpeg | branch: master | Andreas Rheinhardt <andreas.rheinhardt at outlook.com> | Sun Sep 18 20:45:46 2022 +0200| [ce6af2df617f70d6947c7936f9fb462bf5546886] | committer: Andreas Rheinhardt

avcodec/pcm-blurayenc: Don't presume every channel layout to be native

The pcm_bluray encoder has AVCodec.ch_layouts set, so that
ff_encode_preinit() checks that the channel layout in use
is equivalent to one of the layouts from AVCodec.ch_layouts.
Yet equivalent is not the same as identical; in particular,
custom layouts equivalent to native layouts are possible
(and necessary if one wants to use the name/opaque fields
with an ordinary channel layout), so one must not simply
use AVChannelLayout.u.mask. Use av_channel_layout_subset()
instead.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt at outlook.com>

> http://git.videolan.org/gitweb.cgi/ffmpeg.git/?a=commit;h=ce6af2df617f70d6947c7936f9fb462bf5546886
---

 libavcodec/pcm-blurayenc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/libavcodec/pcm-blurayenc.c b/libavcodec/pcm-blurayenc.c
index 1f1a3a27d8..03ed88b8ae 100644
--- a/libavcodec/pcm-blurayenc.c
+++ b/libavcodec/pcm-blurayenc.c
@@ -63,7 +63,7 @@ static av_cold int pcm_bluray_encode_init(AVCodecContext *avctx)
         return AVERROR_BUG;
     }
 
-    switch (avctx->ch_layout.u.mask) {
+    switch (av_channel_layout_subset(&avctx->ch_layout, ~(uint64_t)0)) {
     case AV_CH_LAYOUT_MONO:
         ch_layout = 1;
         break;