[FFmpeg-devel] [PATCH] avcodec/ffv1dec: fix some unsupported pix_fmt

[Jerome Martinez]

FFV1 decoder:

When checking pix_fmt mapping, some bitstreams are mapped to an 
incorrect pix_fmt instead of being rejected (ENOSYS).
Actually, such bitstreams are not supported (FFmpeg encoder does not 
produce such bitstream, such bitstream may come only from another 
encoder for the moment).

- JPEG 2000 RCT 11/13/15/16 bit depths are mapped to a 8-bit FFmpeg 
pix_fmt (e.g. bgr0), which is not expected.
- JPEG 2000 RCT 9/10/12/14 bit depths with alpha are mapped to a FFmpeg 
pix_fmt without alpha (e.g. AV_PIX_FMT_GBRP9 for 9-bit with alpha), 
which is not expected.

The order for choosing the pix_fmt is changed to the one used by YCbCr 
selection (<=8 bit first).
" && !f->transparency" is added to the other lines.

-------------- next part --------------
From 90bfd748b0e25d7a0be037280f4a0a40242f8d27 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?J=C3=A9r=C3=B4me=20Martinez?= <jerome at mediaarea.net>
Date: Mon, 13 Jun 2016 19:18:22 +0200
Subject: [PATCH] avcodec/ffv1dec: fix some unsupported pix_fmt
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Signed-off-by: Jérôme Martinez <jerome at mediaarea.net>
---
 libavcodec/ffv1dec.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/libavcodec/ffv1dec.c b/libavcodec/ffv1dec.c
index d2bf3a8..6a932b2 100644
--- a/libavcodec/ffv1dec.c
+++ b/libavcodec/ffv1dec.c
@@ -768,17 +768,18 @@ static int read_header(FFV1Context *f)
                    "chroma subsampling not supported in this colorspace\n");
             return AVERROR(ENOSYS);
         }
-        if (     f->avctx->bits_per_raw_sample ==  9)
+        if (     f->avctx->bits_per_raw_sample <=  8 && !f->transparency)
+            f->avctx->pix_fmt = AV_PIX_FMT_0RGB32;
+        else if (f->avctx->bits_per_raw_sample <=  8 && f->transparency)
+            f->avctx->pix_fmt = AV_PIX_FMT_RGB32;
+        else if (f->avctx->bits_per_raw_sample ==  9 && !f->transparency)
             f->avctx->pix_fmt = AV_PIX_FMT_GBRP9;
-        else if (f->avctx->bits_per_raw_sample == 10)
+        else if (f->avctx->bits_per_raw_sample == 10 && !f->transparency)
             f->avctx->pix_fmt = AV_PIX_FMT_GBRP10;
-        else if (f->avctx->bits_per_raw_sample == 12)
+        else if (f->avctx->bits_per_raw_sample == 12 && !f->transparency)
             f->avctx->pix_fmt = AV_PIX_FMT_GBRP12;
-        else if (f->avctx->bits_per_raw_sample == 14)
+        else if (f->avctx->bits_per_raw_sample == 14 && !f->transparency)
             f->avctx->pix_fmt = AV_PIX_FMT_GBRP14;
-        else
-        if (f->transparency) f->avctx->pix_fmt = AV_PIX_FMT_RGB32;
-        else                 f->avctx->pix_fmt = AV_PIX_FMT_0RGB32;
     } else {
         av_log(f->avctx, AV_LOG_ERROR, "colorspace not supported\n");
         return AVERROR(ENOSYS);
-- 
2.7.0.windows.1