[FFmpeg-devel] adding RGBA and BGRA to nvenc.c

[Sven C. Dack]

Hello,

I haven't been sending patches in ages. Can somebody walk me through?

The patch is a "low hanging fruit" and simply adds the pixel formats RGBA and 
BGRA to the Nvidia encoder "nvenc", which supports these two formats natively. 
The gain here is that when one grabs the screen with x11grab and chains it 
through the encoder does it no longer require a RGB->YUV conversion and so 
speeds up live streaming (i.e. from 47fp/s to 62fp/s).

Note the twist in AV_PIX_FMT_RGBA versus NV_ENC_BUFFER_FORMAT_ABGR. This is 
intentionally.

Thanks,
Sven

--- a/libavcodec/nvenc.c
+++ b/libavcodec/nvenc.c
@@ -81,6 +81,8 @@ const enum AVPixelFormat ff_nvenc_pix_fmts[] = {
      AV_PIX_FMT_P010,
      AV_PIX_FMT_YUV444P,
      AV_PIX_FMT_YUV444P16,
+    AV_PIX_FMT_RGBA,
+    AV_PIX_FMT_BGRA,
  #if CONFIG_CUDA
      AV_PIX_FMT_CUDA,
  #endif
@@ -1032,6 +1034,14 @@ static av_cold int nvenc_alloc_surface(AVCodecContext 
*avctx, int idx)
          ctx->surfaces[idx].format = NV_ENC_BUFFER_FORMAT_YUV444_10BIT;
          break;

+    case AV_PIX_FMT_RGBA:
+        ctx->surfaces[idx].format = NV_ENC_BUFFER_FORMAT_ABGR;
+        break;
+
+    case AV_PIX_FMT_BGRA:
+        ctx->surfaces[idx].format = NV_ENC_BUFFER_FORMAT_ARGB;
+        break;
+
      default:
          av_log(avctx, AV_LOG_FATAL, "Invalid input pixel format\n");
          return AVERROR(EINVAL);
@@ -1350,6 +1360,14 @@ static int nvenc_copy_frame(AVCodecContext *avctx, 
NvencSurface *inSurf,
          av_image_copy_plane(buf, lockBufferParams->pitch,
              frame->data[2], frame->linesize[2],
              avctx->width << 1, avctx->height);
+    } else if (frame->format == AV_PIX_FMT_RGBA) {
+      av_image_copy_plane(buf, lockBufferParams->pitch,
+           frame->data[0], frame->linesize[0],
+           avctx->width << 2, avctx->height);
+    } else if (frame->format == AV_PIX_FMT_BGRA) {
+      av_image_copy_plane(buf, lockBufferParams->pitch,
+           frame->data[0], frame->linesize[0],
+           avctx->width << 2, avctx->height);
      } else {
          av_log(avctx, AV_LOG_FATAL, "Invalid pixel format!\n");
          return AVERROR(EINVAL);