[FFmpeg-devel] [RFC PATCH 1/2] libavdevice/pipewiregrab: add pipewire based grab
Paul B Mahol
onemda at gmail.com
Thu Sep 21 13:39:45 EEST 2023
On Wed, Sep 20, 2023 at 8:41 PM Abhishek Ojha <
abhishek.ojha at savoirfairelinux.com> wrote:
> This is an proof of concept for pipewire grab to enable screen capture
> on wayland. Add a new Linux capture based on [1] PipeWire and the
> [2] Desktop portal.
> This new capture starts by asking the Desktop portal for a screencapture
> session.There are quite a few D-Bus calls involved in this, but the key
> points are:
>
> 1. A connection to org.freedesktop.portal.ScreenCast is estabilished,
> and the available cursor modes are updated. Currently only embedded
> and hidden currsor mode enabled.
>
> 2. Call CreateSession via dbus call. This is the first step of the
> communication. Response callback return the status of created
> session.
>
> 3. Call SelectSources . This is when a system dialog pops up asking
> the user to either select a monitor (desktop capture).Only monitor
> capture is enabled in current implementation.
>
> 4. Call Start . This signals the compositor that it can setup a PipeWire
> stream, and start sending buffers.
>
> Above flow is implemented as per the [2] xdg-desktop-portal. Once flow
> is completed, pipewire fd is received and using this pipewire stream is
> created and receive buffer from the created stream.
>
> For cursor implementation, embedded cursor mode is enabled that means
> cursor metadata is not handled in current implementation and has no
> control over the cursor bitmap.
>
> gdbus/pipewire logic, this is based on obs-xdg, gstpipewire and
> pipewire examples, and initial pipewire grab logic, this is based on
> libavdevice/xcbgrab and libavdevice/v4l2
>
> This implementation shows the skeleton implementation and enables basic
> functionality. I'd like to hear opinions and suggestions to improve and
> properly use this.
>
> [1] https://pipewire.org/
> [2] https://github.com/flatpak/xdg-desktop-portal/
>
> Below are the arguments for pipewiregrab.
> ffplay -f pipewiregrab -draw_mouse 1 -i :0.0
>
> Signed-off-by: Abhishek Ojha <abhishek.ojha at savoirfairelinux.com>
> ---
> configure | 9 +
> libavdevice/Makefile | 1 +
> libavdevice/alldevices.c | 1 +
> libavdevice/pipewiregrab.c | 1836 ++++++++++++++++++++++++++++++++++++
> 4 files changed, 1847 insertions(+)
> create mode 100644 libavdevice/pipewiregrab.c
>
> diff --git a/configure b/configure
> index e40dcce09e..325b10484f 100755
> --- a/configure
> +++ b/configure
> @@ -299,6 +299,7 @@ External library support:
> --enable-libxcb-shm enable X11 grabbing shm communication
> [autodetect]
> --enable-libxcb-xfixes enable X11 grabbing mouse rendering
> [autodetect]
> --enable-libxcb-shape enable X11 grabbing shape rendering
> [autodetect]
> + --enable-libpipewire enable screen grabbing using pipewire
> [autodetect]
> --enable-libxvid enable Xvid encoding via xvidcore,
> native MPEG-4/Xvid encoder exists [no]
> --enable-libxml2 enable XML parsing using the C library
> libxml2, needed
> @@ -1788,6 +1789,8 @@ EXTERNAL_AUTODETECT_LIBRARY_LIST="
> libxcb_shm
> libxcb_shape
> libxcb_xfixes
> + libpipewire
> + libgio_unix
> lzma
> mediafoundation
> metal
> @@ -3621,6 +3624,7 @@ v4l2_outdev_suggest="libv4l2"
> vfwcap_indev_deps="vfw32 vfwcap_defines"
> xcbgrab_indev_deps="libxcb"
> xcbgrab_indev_suggest="libxcb_shm libxcb_shape libxcb_xfixes"
> +pipewiregrab_indev_deps="libpipewire libgio_unix pthreads"
> xv_outdev_deps="xlib_xv xlib_x11 xlib_xext"
>
> # protocols
> @@ -7041,6 +7045,11 @@ if enabled libxcb; then
> enabled libxcb_xfixes && check_pkg_config libxcb_xfixes xcb-xfixes
> xcb/xfixes.h xcb_xfixes_get_cursor_image
> fi
>
> +enabled libpipewire && check_pkg_config libpipewire "libpipewire-0.3 >=
> 0.3.40" pipewire/pipewire.h pw_init
> +if enabled libpipewire; then
> + enabled libgio_unix && check_pkg_config libgio_unix gio-unix-2.0
> gio/gio.h g_main_loop_new
> +fi
> +
> check_func_headers "windows.h" CreateDIBSection "$gdigrab_indev_extralibs"
>
> # d3d11va requires linking directly to dxgi and d3d11 if not building for
> diff --git a/libavdevice/Makefile b/libavdevice/Makefile
> index c30449201d..f02960782d 100644
> --- a/libavdevice/Makefile
> +++ b/libavdevice/Makefile
> @@ -49,6 +49,7 @@ OBJS-$(CONFIG_V4L2_INDEV) += v4l2.o
> v4l2-common.o timefilter.o
> OBJS-$(CONFIG_V4L2_OUTDEV) += v4l2enc.o v4l2-common.o
> OBJS-$(CONFIG_VFWCAP_INDEV) += vfwcap.o
> OBJS-$(CONFIG_XCBGRAB_INDEV) += xcbgrab.o
> +OBJS-$(CONFIG_PIPEWIREGRAB_INDEV) += pipewiregrab.o
> OBJS-$(CONFIG_XV_OUTDEV) += xv.o
>
> # external libraries
> diff --git a/libavdevice/alldevices.c b/libavdevice/alldevices.c
> index 8a90fcb5d7..1fa8563df4 100644
> --- a/libavdevice/alldevices.c
> +++ b/libavdevice/alldevices.c
> @@ -53,6 +53,7 @@ extern const AVInputFormat ff_v4l2_demuxer;
> extern const FFOutputFormat ff_v4l2_muxer;
> extern const AVInputFormat ff_vfwcap_demuxer;
> extern const AVInputFormat ff_xcbgrab_demuxer;
> +extern const AVInputFormat ff_pipewiregrab_demuxer;
> extern const FFOutputFormat ff_xv_muxer;
>
> /* external libraries */
> diff --git a/libavdevice/pipewiregrab.c b/libavdevice/pipewiregrab.c
> new file mode 100644
> index 0000000000..5b96d43863
> --- /dev/null
> +++ b/libavdevice/pipewiregrab.c
> @@ -0,0 +1,1836 @@
> +/*
> + * PipeWire input grabber (ScreenCast)
> + * Copyright (C) 2023 Savoir-faire Linux, Inc.
> + *
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
> 02110-1301 USA
> + */
> +
> +/**
> + * @file
> + * Pipewire Grab demuxer
> + * @author Abhishek Ojha <abhishek.ojha at savoirfairelinux.com>
> + */
> +
> +#include "config.h"
> +
> +#include <fcntl.h>
> +#include <linux/dma-buf.h>
> +#include <math.h>
> +#include <pthread.h>
> +#include <stdatomic.h>
> +#include <stdlib.h>
> +#include <string.h>
> +#include <sys/mman.h>
> +#include <sys/queue.h>
> +
> +#include "libavutil/internal.h"
> +#include "libavutil/mathematics.h"
> +#include "libavutil/opt.h"
> +#include "libavutil/parseutils.h"
> +#include "libavutil/time.h"
> +
> +#include "libavformat/avformat.h"
> +#include "libavformat/internal.h"
> +
> +#include <pipewire/pipewire.h>
> +#include <pipewire/thread-loop.h>
> +#include <spa/debug/types.h>
> +#include <spa/param/video/format-utils.h>
> +#include <spa/param/video/type-info.h>
> +
> +#include <gio/gio.h>
> +#include <gio/gunixfdlist.h>
> +
> +#ifndef __USE_XOPEN2K8
> +#define F_DUPFD_CLOEXEC
> \
> + 1030 /* Duplicate file descriptor with close-on-exit set. */
> +#endif
> +
> +#define BYTES_PER_PIXEL 4 /* currently all formats assume 4 bytes per
> pixel */
> +#define REQUEST_PATH "/org/freedesktop/portal/desktop/request/%s/obs%u"
> +#define SESSION_PATH "/org/freedesktop/portal/desktop/session/%s/obs%u"
> +#define MAX_SPA_PARAM 4 /* max number of params for spa pod */
> +
> +#define CURSOR_META_SIZE(width, height)
> \
> + (sizeof(struct spa_meta_cursor) + sizeof(struct spa_meta_bitmap) +
> \
> + width * height * 4)
> +
> +/**
> + * Pipewire capture types
> + */
> +typedef enum {
> + DESKTOP_CAPTURE = 1,
> + WINDOW_CAPTURE = 2,
> +} pw_capture_type;
> +
> +/**
> + * Pipewire version structure
> + */
> +struct FFmpegPwVersion {
> + int major;
> + int minor;
> + int micro;
> +};
> +
> +/**
> + * Pipewire structure for frame processing
> + */
> +struct PwStreamAndBuffer {
> + AVFormatContext *ctx;
> + struct pw_stream *pw_stream;
> + struct pw_buffer *pw_buf;
> +};
> +
> +/**
> + * Pipewire supported cursor modes
> + */
> +enum PortalCursorMode {
> + PORTAL_CURSOR_MODE_HIDDEN = 1 << 0,
> + PORTAL_CURSOR_MODE_EMBEDDED = 1 << 1,
> + PORTAL_CURSOR_MODE_METADATA = 1 << 2,
> +};
> +
> +/**
> + * PipeWire Grab main structure
> + * Contains all necessary data that hold current state
> + * Initial state of this struct is allocated by libavdevice
> + * logic when declaring the AVInputFormat ff_pipewiregrab_demuxer.
> + * This structure is priv_data of AVFormatContext instance.
> + */
> +typedef struct PipewireGrabContext {
> + /** thread used to intialize/start pipewire logic */
> + pthread_t pipewire_pthread;
> +
> + /** conditional synchronization logic elecments between pipewire
> + * thread and libavdevice thread.
> + */
> + pthread_cond_t avstream_codec_cond_var;
> + pthread_mutex_t avstream_codec_mutex;
> + atomic_int avstream_codec_flag;
> +
> + pthread_mutex_t current_pkt_mutex;
> + AVPacket* current_pkt;
> +
> + GDBusConnection *connection;
> + GDBusProxy *proxy;
> + GCancellable *cancellable;
> +
> + char *sender_name;
> + char *session_handle;
> +
> + uint32_t pipewire_node;
> + int pipewire_fd;
> +
> + uint32_t available_cursor_modes;
> +
> + GMainLoop *glib_main_loop;
> + struct pw_thread_loop *thread_loop;
> + struct pw_context *context;
> +
> + struct pw_core *core;
> + struct spa_hook core_listener;
> +
> + struct pw_stream *stream;
> + struct spa_hook stream_listener;
> + struct spa_video_info format;
> +
> + pw_capture_type capture_type;
> + bool negotiated;
> +
> + /**< draw_mouse is to control the enable/disable mouse cursor */
> + int draw_mouse;
> +
> + /** cursor metadata */
> + struct {
> + bool visible;
> + bool valid;
> + int x, y;
> + int hotspot_x, hotspot_y;
> + int width, height;
> + } cursor;
> +
> + /**< Width and height of the grab frame (private option) */
> + uint32_t width, height;
> +
> + /**< Size in bytes of the frame pixel data */
> + uint32_t frame_size;
> + uint8_t Bpp;
> + enum AVPixelFormat av_pxl_format;
> + AVRational user_frame_rate;
> +
> + int64_t time_frame;
> + AVRational time_base;
> + int64_t frame_duration;
> +
> + const AVClass *class;
> +
> + const char *framerate;
> + struct FFmpegPwVersion server_version;
> + int server_version_sync;
> +} PipewireGrabContext;
> +
> +/**
> + * dbus's method/event marshalling structure
> + */
> +struct DbusCallData {
> + AVFormatContext *ctx;
> + char *request_path;
> + guint signal_id;
> + gulong cancelled_id;
> +};
> +
> +#define FOLLOW_CENTER -1
> +
> +#define OFFSET(x) offsetof(PipewireGrabContext, x)
> +#define D AV_OPT_FLAG_DECODING_PARAM
> +static const AVOption options[] = {
> + { "framerate",
> + "",
> + OFFSET(framerate),
> + AV_OPT_TYPE_STRING,
> + { .str = "ntsc" },
> + 0,
> + 0,
> + D },
> + { "draw_mouse",
> + "Draw the mouse pointer.",
> + OFFSET(draw_mouse),
> + AV_OPT_TYPE_INT,
> + { .i64 = 0 },
> + 0,
> + 1,
> + D },
> + { NULL },
> +};
> +
> +/**
> + * Function parse the pipewire version
> + * @param dst FmpegPwVersion that contains PipeWire version info
> + * @param version pipewire version
> + */
> +static bool parse_pw_version(struct FFmpegPwVersion *dst, const char
> *version)
> +{
> + int n_matches = sscanf(version, "%d.%d.%d", &dst->major, &dst->minor,
> + &dst->micro);
> + return n_matches == 3;
> +}
> +
> +/**
> + * Function update the pipewire version in private data
> + * params, then signal the blocked @ref pipewiregrab_read_header
> + *
> + * @param ctx AVFormatContext that contains PipeWire Grab main structure
> + * @param version pipewire version
> + */
> +static void update_pw_versions(AVFormatContext *ctx, const char *version)
> +{
> + PipewireGrabContext *pw_ctx = ctx->priv_data;
> + av_log(ctx, AV_LOG_DEBUG, "[pipewiregrab] Server version: %s\n",
> version);
>
All logs appear to have this [pipewiregrab] part, but this looks unneeded
as logs normally write module name out.
More information about the ffmpeg-devel
mailing list