[FFmpeg-devel] [PATCH 3/3] avcodec/nvenc: Declare support for P016

[Philip Langdale]

nvenc doesn't support P016, but we have two problems today:

1) We declare support for YUV444P16 which nvenc also doesn't support.
   We do this because it's the only pix_fmt we have that can
   approximate nvenc's internal format that is YUV444P10 with data in
   MSBs instead of LSBs. Because the declared format is a 16bit one,
   it will be preferrentially chosen when encoding >10bit content,
   but that content will normally be YUV420P12 or P016 which should
   get mapped to P010 and not YUV444P10.

2) Transcoding P016 content with nvenc should be possible in a pure
   hardware pipeline, and that can't be done if nvenc doesn't say it
   accepts P016. By mapping it to P010, we can use it, albeit with
   truncation. I have established that swscale doesn't know how to
   dither to 10bits so we'd get truncation anyway, even if we tried
   to do this 'properly'.

Signed-off-by: Philip Langdale <philipl at overt.org>
---
 libavcodec/nvenc.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
index 161c56adc2..e00c1fb245 100644
--- a/libavcodec/nvenc.c
+++ b/libavcodec/nvenc.c
@@ -41,7 +41,8 @@ const enum AVPixelFormat ff_nvenc_pix_fmts[] = {
     AV_PIX_FMT_NV12,
     AV_PIX_FMT_P010,
     AV_PIX_FMT_YUV444P,
-    AV_PIX_FMT_YUV444P16,
+    AV_PIX_FMT_P016,      // Truncated to 10bits
+    AV_PIX_FMT_YUV444P16, // Truncated to 10bits
     AV_PIX_FMT_0RGB32,
     AV_PIX_FMT_0BGR32,
     AV_PIX_FMT_CUDA,
@@ -52,6 +53,7 @@ const enum AVPixelFormat ff_nvenc_pix_fmts[] = {
 };
 
 #define IS_10BIT(pix_fmt)  (pix_fmt == AV_PIX_FMT_P010    || \
+                            pix_fmt == AV_PIX_FMT_P016    || \
                             pix_fmt == AV_PIX_FMT_YUV444P16)
 
 #define IS_YUV444(pix_fmt) (pix_fmt == AV_PIX_FMT_YUV444P || \
@@ -1222,6 +1224,7 @@ static NV_ENC_BUFFER_FORMAT nvenc_map_buffer_format(enum AVPixelFormat pix_fmt)
     case AV_PIX_FMT_NV12:
         return NV_ENC_BUFFER_FORMAT_NV12_PL;
     case AV_PIX_FMT_P010:
+    case AV_PIX_FMT_P016:
         return NV_ENC_BUFFER_FORMAT_YUV420_10BIT;
     case AV_PIX_FMT_YUV444P:
         return NV_ENC_BUFFER_FORMAT_YUV444_PL;
-- 
2.14.1