[FFmpeg-devel] [PATCH] Add android_capture indev
Nicolas George
george at nsup.org
Thu Nov 2 15:40:37 EET 2017
Le duodi 12 brumaire, an CCXXVI, Felix Matouschek a écrit :
> I've written an indev for Android devices to allow capturing their builtin
> cameras.
> What needs to be done to merge this?
Thanks for the patch. It looks very interesting. Before considering
merging, there are a few technical point to address. See inline comments
below.
> From b21fc8729ef2e1d9867dd7652f2c6173378e4910 Mon Sep 17 00:00:00 2001
> From: Felix Matouschek <felix at matouschek.org>
> Date: Tue, 24 Oct 2017 13:11:23 +0200
> Subject: [PATCH] Add android_capture indev
> To: ffmpeg-devel at ffmpeg.org
>
> This commit adds an indev for Android devices on API level 24+ which
> uses the Android NDK Camera2 API to capture video from builtin cameras
>
> Signed-off-by: Felix Matouschek <felix at matouschek.org>
> ---
> configure | 6 +
> libavdevice/Makefile | 1 +
> libavdevice/alldevices.c | 1 +
> libavdevice/android_capture.c | 782 ++++++++++++++++++++++++++++++++++++++++++
When reading the subject of the mail, I first thought it would be about
screen capture. Furthermore, there is code for audio, but it is not
connected to anything, and it does not seem that the android API
connects audio and video together.
For all these reasons, I suggest you name this device maybe
android_camera, and keep android_mic for audio capture.
> libavdevice/android_capture.h | 77 +++++
This header is only used once, it does not need to be separate.
> 5 files changed, 867 insertions(+)
> create mode 100644 libavdevice/android_capture.c
> create mode 100644 libavdevice/android_capture.h
>
> diff --git a/configure b/configure
> index 7a53bc76c7..e2165f2ff9 100755
> --- a/configure
> +++ b/configure
> @@ -3068,6 +3068,8 @@ xmv_demuxer_select="riffdec"
> xwma_demuxer_select="riffdec"
>
> # indevs / outdevs
> +android_capture_indev_deps="android mediandk camera2ndk pthreads"
> +android_capture_indev_extralibs="-landroid -lmediandk -lcamera2ndk"
> alsa_indev_deps="alsa"
> alsa_outdev_deps="alsa"
> avfoundation_indev_deps="avfoundation corevideo coremedia pthreads"
> @@ -5836,6 +5838,10 @@ check_lib shell32 "windows.h shellapi.h" CommandLineToArgvW -lshell32
> check_lib wincrypt "windows.h wincrypt.h" CryptGenRandom -ladvapi32
> check_lib psapi "windows.h psapi.h" GetProcessMemoryInfo -lpsapi
>
> +check_lib android android/native_window.h ANativeWindow_acquire -landroid
> +check_lib mediandk "stdint.h media/NdkImage.h" AImage_delete -lmediandk
> +check_lib camera2ndk "stdbool.h stdint.h camera/NdkCameraManager.h" ACameraManager_create -lcamera2ndk
> +
> enabled appkit && check_apple_framework AppKit
> enabled audiotoolbox && check_apple_framework AudioToolbox
> enabled avfoundation && check_apple_framework AVFoundation
> diff --git a/libavdevice/Makefile b/libavdevice/Makefile
> index 8228d62147..aa01dd7e24 100644
> --- a/libavdevice/Makefile
> +++ b/libavdevice/Makefile
> @@ -14,6 +14,7 @@ OBJS-$(CONFIG_SHARED) += reverse.o
> # input/output devices
> OBJS-$(CONFIG_ALSA_INDEV) += alsa_dec.o alsa.o timefilter.o
> OBJS-$(CONFIG_ALSA_OUTDEV) += alsa_enc.o alsa.o
> +OBJS-$(CONFIG_ANDROID_CAPTURE_INDEV) += android_capture.o
> OBJS-$(CONFIG_AVFOUNDATION_INDEV) += avfoundation.o
> OBJS-$(CONFIG_BKTR_INDEV) += bktr.o
> OBJS-$(CONFIG_CACA_OUTDEV) += caca.o
> diff --git a/libavdevice/alldevices.c b/libavdevice/alldevices.c
> index b767b6a718..6cd57aa88a 100644
> --- a/libavdevice/alldevices.c
> +++ b/libavdevice/alldevices.c
> @@ -42,6 +42,7 @@ static void register_all(void)
> {
> /* devices */
> REGISTER_INOUTDEV(ALSA, alsa);
> + REGISTER_INDEV (ANDROID_CAPTURE, android_capture);
> REGISTER_INDEV (AVFOUNDATION, avfoundation);
> REGISTER_INDEV (BKTR, bktr);
> REGISTER_OUTDEV (CACA, caca);
> diff --git a/libavdevice/android_capture.c b/libavdevice/android_capture.c
> new file mode 100644
> index 0000000000..be0dee8f81
> --- /dev/null
> +++ b/libavdevice/android_capture.c
> @@ -0,0 +1,782 @@
> +/*
> + * Android camera/microphone input source via Android NDK APIs (Audio yet to be implemented)
> + *
> + * Copyright (C) 2017 Felix Matouschek
> + *
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#include <errno.h>
> +#include <stdbool.h>
> +#include <stdint.h>
> +
> +#include <camera/NdkCameraDevice.h>
> +#include <camera/NdkCameraManager.h>
> +#include <media/NdkImage.h>
> +
> +#include "libavformat/internal.h"
> +#include "libavutil/avstring.h"
> +#include "libavutil/display.h"
> +#include "libavutil/imgutils.h"
> +#include "libavutil/opt.h"
> +#include "libavutil/parseutils.h"
> +
> +#include "android_capture.h"
> +#include "version.h"
> +
> +/* This image format is available on all Android devices
> + * supporting the Camera2 API */
> +#define IMAGE_FORMAT_ANDROID AIMAGE_FORMAT_YUV_420_888
> +#define IMAGE_FORMAT_FFMPEG AV_PIX_FMT_YUV420P
> +#define IMAGE_NUM_PLANES 3
> +
> +#define MAX_BUF_COUNT 2
> +#define VIDEO_STREAM_ID 0
> +#define VIDEO_TIMEBASE 1000000000
> +#define AUDIO_STREAM_ID 1
> +
> +static int parse_token_to_int(char *token, int *out)
> +{
> + long l;
> + char *endptr;
> + errno = 0;
> +
> + l = strtol(token, &endptr, 0);
> +
> + if (errno == ERANGE || *endptr != '\0' || token == endptr) {
> + return AVERROR(ERANGE);
The second and third clauses should return EINVAL.
> + }
> +
> +#if LONG_MIN < INT_MIN || LONG_MAX > INT_MAX
> + if (l < INT_MIN || l > INT_MAX) {
> + return AVERROR(ERANGE);
> + }
> +#endif
> +
> + *out = (int) l;
> +
> + return 0;
> +}
This does not belong in an individual component. And I think it is not
needed at all, see below.
> +
> +static int parse_device_number(AVFormatContext *avctx)
> +{
> + android_capture_ctx *ctx = avctx->priv_data;
> + int *video_device_number = &ctx->video_device_number;
> + int *audio_device_number = &ctx->audio_device_number;
These extra variable seem quite unnecessary.
> + char *name = av_strdup(avctx->filename);
> + char *tmp = name;
> + char *type;
> + char *save;
> + int ret = 0;
> +
> + *video_device_number = -1;
> + *audio_device_number = -1;
> +
> + while ((type = av_strtok(tmp, "=", &save))) {
> + char *token = av_strtok(NULL, ":", &save);
> + tmp = NULL;
You are parsing the "filename" of the device into a key=value syntax.
This is not a good idea, and I really would not like another key=value
parser in the code base.
How do cameraIds look? If the string values are friendly enough, they
can be used as filenames as is.
> +
> + if (!strcmp(type, "video")) {
> + ret = parse_token_to_int(token, video_device_number);
> + if (ret < 0) {
> + break;
> + }
> +
> + } else if (!strcmp(type, "audio")) {
> + ret = parse_token_to_int(token, audio_device_number);
> + if (ret < 0) {
> + break;
> + }
> + } else {
> + ret = AVERROR(ENXIO);
The correct return code would be EINVAL in this case.
> + break;
> + }
> + }
> +
> + if (ret >= 0 && *video_device_number < 0 && *audio_device_number < 0) {
> + ret = AVERROR(ENXIO);
EINVAL here too.
> + }
> +
> + av_free(name);
> + return ret;
> +}
> +
> +static void camera_dev_disconnected(void *context, ACameraDevice *device)
> +{
> + AVFormatContext *avctx = (AVFormatContext *) context;
The cast is unnecessary, and as such harmful.
> + android_capture_ctx *ctx = avctx->priv_data;
> + ctx->read_closing = 1;
If I read the code correctly, Android is expected to call this from a
separate thread. Then this needs to be protected by a memory barrier,
mutex or other.
> + av_log(avctx, AV_LOG_ERROR, "Camera with id %s disconnected.\n",
> + ACameraDevice_getId(device));
> +}
> +
> +static void camera_dev_error(void *context, ACameraDevice *device, int error)
> +{
> + AVFormatContext *avctx = (AVFormatContext *) context;
Same as above.
> + android_capture_ctx *ctx = avctx->priv_data;
> + ctx->read_closing = 1;
Same as above.
> + av_log(avctx, AV_LOG_ERROR, "Error %d on camera with id %s.\n", error,
> + ACameraDevice_getId(device));
Can the error be translated into a human-readable message?
> +}
> +
> +static int open_camera(AVFormatContext *avctx)
> +{
> + android_capture_ctx *ctx = avctx->priv_data;
> + camera_status_t ret;
> + ACameraIdList *camera_ids;
> +
> + ret = ACameraManager_getCameraIdList(ctx->camera_mgr, &camera_ids);
> + if (ret != ACAMERA_OK) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to get camera id list, camera_status_t: %d.\n", ret);
> + return AVERROR(EIO);
Better translate the ACAMERA codes into AVERROR codes, or use
AVERROR_EXTERNAL. Same below.
> + }
> +
> + if (ctx->video_device_number < camera_ids->numCameras) {
> + ctx->camera_id = av_strdup(
> + camera_ids->cameraIds[ctx->video_device_number]);
> + } else {
> + av_log(avctx, AV_LOG_ERROR, "No camera with number %d available.\n",
> + ctx->video_device_number);
> + return AVERROR(ENXIO);
> + }
> +
> + ACameraManager_deleteCameraIdList(camera_ids);
> +
> + ret = ACameraManager_getCameraCharacteristics(ctx->camera_mgr,
> + ctx->camera_id, &ctx->camera_metadata);
> + if (ret != ACAMERA_OK) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to get metadata for camera with id %s, camera_status_t: %d.\n",
> + ctx->camera_id, ret);
> + return AVERROR(EIO);
> + }
> +
> + ctx->camera_state_callbacks.context = avctx;
> + ctx->camera_state_callbacks.onDisconnected = camera_dev_disconnected;
> + ctx->camera_state_callbacks.onError = camera_dev_error;
> +
> + ret = ACameraManager_openCamera(ctx->camera_mgr, ctx->camera_id,
> + &ctx->camera_state_callbacks, &ctx->camera_dev);
> + if (ret != ACAMERA_OK) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to open camera with id %s, camera_status_t: %d.\n",
> + ctx->camera_id, ret);
> + return AVERROR(EIO);
> + }
> +
> + return 0;
> +}
> +
> +static void get_sensor_orientation(AVFormatContext *avctx)
> +{
> + android_capture_ctx *ctx = avctx->priv_data;
> + ACameraMetadata_const_entry lens_facing;
> + ACameraMetadata_const_entry sensor_orientation;
> +
> + ACameraMetadata_getConstEntry(ctx->camera_metadata, ACAMERA_LENS_FACING,
> + &lens_facing);
> + ACameraMetadata_getConstEntry(ctx->camera_metadata,
> + ACAMERA_SENSOR_ORIENTATION, &sensor_orientation);
> +
> + ctx->lens_facing = lens_facing.data.u8[0];
> + ctx->sensor_orientation = sensor_orientation.data.i32[0];
> +}
> +
> +static int match_video_size(AVFormatContext *avctx)
> +{
> + android_capture_ctx *ctx = avctx->priv_data;
> + ACameraMetadata_const_entry available_configs;
> + int ret = -1;
> +
> + ACameraMetadata_getConstEntry(ctx->camera_metadata,
> + ACAMERA_SCALER_AVAILABLE_STREAM_CONFIGURATIONS, &available_configs);
> +
> + for (int i = 0; i < available_configs.count; i++) {
> + int32_t input = available_configs.data.i32[i * 4 + 3];
> + int32_t format = available_configs.data.i32[i * 4 + 0];
Is it really the official way of getting the resolution and format of
the video? If so, were the Android people drunk?
> +
> + if (input) {
> + continue;
> + }
> +
> + if (format == IMAGE_FORMAT_ANDROID) {
It would be better to support as many pixel formats as possible.
> + int32_t width = available_configs.data.i32[i * 4 + 1];
> + int32_t height = available_configs.data.i32[i * 4 + 2];
> +
> + //Same ratio
> + if (ctx->requested_width * ctx->requested_height
> + == width * height) {
> + ctx->width = width;
> + ctx->height = height;
> + ret = 0;
> + break;
> + }
> + }
> + }
> +
> + if (ret < 0) {
> + ctx->width = 640;
> + ctx->height = 480;
> + }
> +
> + return ret;
> +}
> +
> +static int match_framerate(AVFormatContext *avctx)
> +{
> + android_capture_ctx *ctx = avctx->priv_data;
> + ACameraMetadata_const_entry available_framerates;
> + int ret = -1;
> + int current_best_match = -1;
> +
> + ACameraMetadata_getConstEntry(ctx->camera_metadata,
> + ACAMERA_CONTROL_AE_AVAILABLE_TARGET_FPS_RANGES,
> + &available_framerates);
> +
> + for (int i = 0; i < available_framerates.count; i++) {
> + int32_t min = available_framerates.data.i32[i * 2 + 0];
> + int32_t max = available_framerates.data.i32[i * 2 + 1];
> + double requested_framerate = av_q2d(ctx->requested_framerate);
> +
> + if (requested_framerate == max) {
> + ret = 0;
> +
> + if (min == max) {
> + ctx->framerate_range[0] = min;
> + ctx->framerate_range[1] = max;
> + break;
> + } else if (current_best_match >= 0) {
> + int32_t current_best_match_min =
> + available_framerates.data.i32[current_best_match * 2 + 0];
> + if (min > current_best_match_min) {
> + current_best_match = i;
> + }
> + } else {
> + current_best_match = i;
> + }
> + }
> + }
> +
> + if (ret < 0) {
> + ctx->framerate_range[0] = available_framerates.data.i32[0];
> + ctx->framerate_range[1] = available_framerates.data.i32[1];
> + }
> +
> + return ret;
> +}
> +
> +static int shall_we_drop(AVFormatContext *avctx, int stream_index,
> + int stream_id)
> +{
> + android_capture_ctx *ctx = avctx->priv_data;
> + static const uint8_t dropscore[] = { 62, 75, 87, 100 };
> + const int ndropscores = FF_ARRAY_ELEMS(dropscore);
> + const char *stream_name = stream_id ? "audio" : "video";
> + unsigned int buffer_fullness = (ctx->curbufsize[stream_index] * 100)
> + / avctx->max_picture_buffer;
> +
> + if (dropscore[++ctx->video_frame_num % ndropscores] <= buffer_fullness) {
> + av_log(avctx, AV_LOG_ERROR,
> + "real-time buffer of [%s input] too full or near too full (%d%% of size: %d [rtbufsize parameter])! frame dropped!\n",
> + stream_name, buffer_fullness, avctx->max_picture_buffer);
> + return 1;
> + }
> +
> + return 0;
> +}
> +
> +static void image_available(void *context, AImageReader *reader)
> +{
> + AVFormatContext *avctx = (AVFormatContext *) context;
> + android_capture_ctx *ctx = avctx->priv_data;
> + media_status_t ret;
> +
> + AImage *image;
> + int64_t image_timestamp;
> + uint8_t *image_plane_data[4];
> + int32_t image_linestrides[4];
> + int plane_data_length[4];
> +
> + AVPacketList *pktl_next;
> + AVPacketList **ppktl = &ctx->pktl;
> + uint8_t *side_data;
> + int pkt_buffer_size = av_image_get_buffer_size(IMAGE_FORMAT_FFMPEG,
> + ctx->width, ctx->height, 1);
> + int32_t display_matrix[9];
> +
> + av_display_rotation_set(display_matrix, 360 - ctx->sensor_orientation);
> +
> + if (ctx->lens_facing == ACAMERA_LENS_FACING_FRONT) {
> + av_display_matrix_flip(display_matrix, 1, 0);
> + }
> +
> + ret = AImageReader_acquireLatestImage(reader, &image);
> + if (ret != AMEDIA_OK) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to acquire latest image from image reader, media_status_t: %d.\n",
> + ret);
Strange indentation.
> + goto error;
> + }
> +
> + // Silently drop frames when read_closing is set
> + if (ctx->read_closing) {
> + goto error;
> + }
> +
> + AImage_getTimestamp(image, &image_timestamp);
> +
> + for (int i = 0; i < IMAGE_NUM_PLANES; i++) {
> + AImage_getPlaneRowStride(image, i, &image_linestrides[i]);
> + AImage_getPlaneData(image, i, &image_plane_data[i], &plane_data_length[i]);
> + }
> +
> + pthread_mutex_lock(&ctx->mutex);
> +
> + if (shall_we_drop(avctx, ctx->video_stream_index, VIDEO_STREAM_ID)) {
> + goto error;
> + }
> +
> + pktl_next = av_mallocz(sizeof(AVPacketList));
I think it would better to avoid malloc() and anything similar in a
critical section. Do all the heavy work first, then enter the critical
section.
> + if (!pktl_next)
> + goto error;
> +
> + if (av_new_packet(&pktl_next->pkt, pkt_buffer_size) < 0) {
> + av_free(pktl_next);
> + goto error;
> + }
> +
> + side_data = av_packet_new_side_data(&pktl_next->pkt,
> + AV_PKT_DATA_DISPLAYMATRIX, sizeof(display_matrix));
It would probably be best to avoid sending the side data repeatedly if
it does not change.
> + if (!side_data) {
> + av_packet_unref(&pktl_next->pkt);
> + av_free(pktl_next);
> + goto error;
> + }
> +
> + pktl_next->pkt.stream_index = ctx->video_stream_index;
> + pktl_next->pkt.pts = image_timestamp;
> + av_image_copy_to_buffer(pktl_next->pkt.data, pkt_buffer_size,
> + (const uint8_t * const *) image_plane_data, image_linestrides,
> + IMAGE_FORMAT_FFMPEG, ctx->width, ctx->height, 1);
> + memcpy(side_data, display_matrix, sizeof(display_matrix));
> +
> + while (*ppktl) {
> + ppktl = &(*ppktl)->next;
> + }
> + *ppktl = pktl_next;
> +
> + ctx->curbufsize[ctx->video_stream_index] += pkt_buffer_size;
> +
> +error:
> + pthread_mutex_unlock(&ctx->mutex);
> + AImage_delete(image);
> +
> + return;
It seems error conditions are not taken into account. Is it on purpose?
> +}
> +
> +static int create_image_reader(AVFormatContext *avctx)
> +{
> + android_capture_ctx *ctx = avctx->priv_data;
> + media_status_t ret;
> +
> + ret = AImageReader_new(ctx->width, ctx->height, IMAGE_FORMAT_ANDROID,
> + MAX_BUF_COUNT, &ctx->image_reader);
> + if (ret != AMEDIA_OK) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to create image reader, media_status_t: %d.\n", ret);
> + return AVERROR(EIO);
> + }
> +
> + ctx->image_listener.context = avctx;
> + ctx->image_listener.onImageAvailable = image_available;
> +
> + ret = AImageReader_setImageListener(ctx->image_reader,
> + &ctx->image_listener);
> + if (ret != AMEDIA_OK) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to set image listener on image reader, media_status_t: %d.\n",
> + ret);
> + return AVERROR(EIO);
> + }
> +
> + ret = AImageReader_getWindow(ctx->image_reader, &ctx->image_reader_window);
> + if (ret != AMEDIA_OK) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Could not get image reader window, media_status_t: %d.\n",
> + ret);
> + return AVERROR(EIO);
> + }
> +
> + return 0;
> +}
> +
> +static void capture_session_closed(void *context,
> + ACameraCaptureSession *session)
> +{
> + av_log((AVFormatContext *) context, AV_LOG_INFO,
The cast is wrong. Same below.
> + "Android camera capture session was closed.\n");
> +}
> +
> +static void capture_session_ready(void *context, ACameraCaptureSession *session)
> +{
> + av_log((AVFormatContext *) context, AV_LOG_INFO,
> + "Android camera capture session is ready.\n");
> +}
> +
> +static void capture_session_active(void *context,
> + ACameraCaptureSession *session)
> +{
> + av_log((AVFormatContext *) context, AV_LOG_INFO,
> + "Android camera capture session is active.\n");
> +}
> +
> +static int create_capture_session(AVFormatContext *avctx)
> +{
> + android_capture_ctx *ctx = avctx->priv_data;
> + camera_status_t ret;
> +
> + ret = ACaptureSessionOutputContainer_create(
> + &ctx->capture_session_output_container);
> + if (ret != ACAMERA_OK) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to create capture session output container, camera_status_t: %d.\n",
> + ret);
> + return AVERROR(EIO);
> + }
> +
> + ANativeWindow_acquire(ctx->image_reader_window);
> +
> + ret = ACaptureSessionOutput_create(ctx->image_reader_window,
> + &ctx->capture_session_output);
> + if (ret != ACAMERA_OK) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to create capture session container, camera_status_t: %d.\n",
> + ret);
> + return AVERROR(EIO);
> + }
> +
> + ret = ACaptureSessionOutputContainer_add(
> + ctx->capture_session_output_container, ctx->capture_session_output);
> + if (ret != ACAMERA_OK) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to add output to output container, camera_status_t: %d.\n",
> + ret);
> + return AVERROR(EIO);
> + }
> +
> + ret = ACameraOutputTarget_create(ctx->image_reader_window,
> + &ctx->camera_output_target);
> + if (ret != ACAMERA_OK) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to create camera output target, camera_status_t: %d.\n",
> + ret);
> + return AVERROR(EIO);
> + }
> +
> + ret = ACameraDevice_createCaptureRequest(ctx->camera_dev, TEMPLATE_RECORD,
> + &ctx->capture_request);
> + if (ret != ACAMERA_OK) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to create capture request, camera_status_t: %d.\n",
> + ret);
> + return AVERROR(EIO);
> + }
> +
> + ret = ACaptureRequest_setEntry_i32(ctx->capture_request,
> + ACAMERA_CONTROL_AE_TARGET_FPS_RANGE, 2, ctx->framerate_range);
> + if (ret != ACAMERA_OK) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to set target fps range in capture request, camera_status_t: %d.\n",
> + ret);
> + return AVERROR(EIO);
> + }
> +
> + ret = ACaptureRequest_addTarget(ctx->capture_request,
> + ctx->camera_output_target);
> + if (ret != ACAMERA_OK) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to add capture request capture request, camera_status_t: %d.\n",
> + ret);
> + return AVERROR(EIO);
> + }
> +
> + ctx->capture_session_state_callbacks.context = avctx;
> + ctx->capture_session_state_callbacks.onClosed = capture_session_closed;
> + ctx->capture_session_state_callbacks.onReady = capture_session_ready;
> + ctx->capture_session_state_callbacks.onActive = capture_session_active;
> +
> + ret = ACameraDevice_createCaptureSession(ctx->camera_dev,
> + ctx->capture_session_output_container,
> + &ctx->capture_session_state_callbacks, &ctx->capture_session);
> + if (ret != ACAMERA_OK) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to create capture session, camera_status_t: %d.\n",
> + ret);
> + return AVERROR(EIO);
> + }
> +
> + ret = ACameraCaptureSession_setRepeatingRequest(ctx->capture_session, NULL,
> + 1, &ctx->capture_request, NULL);
> + if (ret != ACAMERA_OK) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to set repeating request on capture session, camera_status_t: %d.\n",
> + ret);
> + return AVERROR(EIO);
> + }
> +
> + return 0;
> +}
> +
> +static int add_video_stream(AVFormatContext *avctx)
> +{
> + android_capture_ctx *ctx = avctx->priv_data;
> + AVStream *st;
> + AVCodecParameters *codecpar;
> +
> + st = avformat_new_stream(avctx, NULL);
> + if (!st) {
> + return AVERROR(ENOMEM);
> + }
> +
> + st->id = VIDEO_STREAM_ID;
> + st->avg_frame_rate = (AVRational) { ctx->framerate_range[1], 1 };
> + st->r_frame_rate = (AVRational) { ctx->framerate_range[1], 1 };
> +
> + codecpar = st->codecpar;
> + codecpar->codec_type = AVMEDIA_TYPE_VIDEO;
> + codecpar->codec_id = AV_CODEC_ID_RAWVIDEO;
> + codecpar->format = IMAGE_FORMAT_FFMPEG;
> + codecpar->width = ctx->width;
> + codecpar->height = ctx->height;
> +
> + avpriv_set_pts_info(st, 64, 1, VIDEO_TIMEBASE);
> +
> + ctx->video_stream_index = st->index;
> +
> + return 0;
> +}
> +
> +static int android_capture_read_close(AVFormatContext *avctx)
> +{
> + android_capture_ctx *ctx = avctx->priv_data;
> + AVPacketList *pktl;
> +
> + ctx->read_closing = 1;
> +
> + if (ctx->capture_session) {
> + ACameraCaptureSession_stopRepeating(ctx->capture_session);
> + // Following warning is emitted, after capture session closed callback is received:
> + // ACameraCaptureSession: Device is closed but session 0 is not notified
> + // Seems to be a bug in Android, we can ignore this
> + ACameraCaptureSession_close(ctx->capture_session);
> + ctx->capture_session = NULL;
> + }
> +
> + if (ctx->capture_request) {
> + ACaptureRequest_removeTarget(ctx->capture_request,
> + ctx->camera_output_target);
> + ACaptureRequest_free(ctx->capture_request);
> + ctx->capture_request = NULL;
> + }
> +
> + if (ctx->camera_output_target) {
> + ACameraOutputTarget_free(ctx->camera_output_target);
> + ctx->camera_output_target = NULL;
> + }
> +
> + if (ctx->capture_session_output) {
> + ACaptureSessionOutputContainer_remove(
> + ctx->capture_session_output_container,
> + ctx->capture_session_output);
> + ACaptureSessionOutput_free(ctx->capture_session_output);
> + ctx->capture_session_output = NULL;
> + }
> +
> + ANativeWindow_release(ctx->image_reader_window);
> +
> + if (ctx->capture_session_output_container) {
> + ACaptureSessionOutputContainer_free(
> + ctx->capture_session_output_container);
> + ctx->capture_session_output_container = NULL;
> + }
> +
> + if (ctx->camera_dev) {
> + ACameraDevice_close(ctx->camera_dev);
> + ctx->camera_dev = NULL;
> + }
> +
> + if (ctx->image_reader) {
> + AImageReader_delete(ctx->image_reader);
> + ctx->image_reader = NULL;
> + ctx->image_reader_window = NULL;
> + }
> +
> + if (ctx->camera_metadata) {
> + ACameraMetadata_free(ctx->camera_metadata);
> + ctx->camera_metadata = NULL;
> + }
> +
> + if (ctx->camera_id) {
> + av_free(ctx->camera_id);
> + ctx->camera_id = NULL;
av_freep(&ctx->camera_id)
> + }
> +
> + if (ctx->camera_mgr) {
> + ACameraManager_delete(ctx->camera_mgr);
> + ctx->camera_mgr = NULL;
> + }
> +
> + pktl = ctx->pktl;
> + while (pktl) {
> + AVPacketList *next = pktl->next;
> + av_packet_unref(&pktl->pkt);
> + av_free(pktl);
> + pktl = next;
> + }
> +
> + pthread_mutex_destroy(&ctx->mutex);
> +
> + return 0;
> +}
> +
> +static int android_capture_read_header(AVFormatContext *avctx)
> +{
> + android_capture_ctx *ctx = avctx->priv_data;
> + int ret;
> +
> + pthread_mutex_init(&ctx->mutex, 0);
> +
> + ret = parse_device_number(avctx);
> + if (ret < 0) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Malformed android_capture input string.\n");
> + goto error;
> + }
> +
> + if (ctx->framerate) {
> + ret = av_parse_video_rate(&ctx->requested_framerate, ctx->framerate);
> + if (ret < 0) {
> + av_log(avctx, AV_LOG_ERROR, "Could not parse framerate '%s'.\n",
> + ctx->framerate);
> + goto error;
> + }
> + }
> +
> + ctx->camera_mgr = ACameraManager_create();
> + if (!ctx->camera_mgr) {
> + av_log(avctx, AV_LOG_ERROR,
> + "Failed to create Android camera manager.\n");
> + ret = AVERROR(EIO);
> + goto error;
> + }
> +
> + ret = open_camera(avctx);
> + if (ret < 0) {
> + av_log(avctx, AV_LOG_ERROR, "Failed to open Android camera.\n");
> + goto error;
> + }
> +
> + get_sensor_orientation(avctx);
> +
> + if (match_video_size(avctx) < 0) {
> + av_log(avctx, AV_LOG_WARNING,
> + "Requested video_size not available, falling back to 640x480\n");
> + }
> +
> + if (match_framerate(avctx) < 0) {
> + av_log(avctx, AV_LOG_WARNING,
> + "Requested framerate not available, falling back to min: %d and max: %d fps\n",
> + ctx->framerate_range[0], ctx->framerate_range[1]);
> + }
> +
> + ret = create_image_reader(avctx);
> + if (ret < 0) {
> + goto error;
> + }
> +
> + ret = create_capture_session(avctx);
> + if (ret < 0) {
> + goto error;
> + }
> +
> + ret = add_video_stream(avctx);
> +
> +error:
> + if (ret < 0) {
> + android_capture_read_close(avctx);
> + av_log(avctx, AV_LOG_ERROR, "Failed to open android_capture.\n");
> + }
> +
> + return ret;
> +}
> +
> +static int android_capture_read_packet(AVFormatContext *avctx, AVPacket *pkt)
> +{
> + android_capture_ctx *ctx = avctx->priv_data;
> + AVPacketList *pktl = NULL;
> +
> + while (!ctx->read_closing && !pktl) {
> + pthread_mutex_lock(&ctx->mutex);
> + pktl = ctx->pktl;
> + if (pktl) {
> + *pkt = pktl->pkt;
> + ctx->pktl = ctx->pktl->next;
> + av_free(pktl);
> + ctx->curbufsize[pkt->stream_index] -= pkt->size;
> + }
> + pthread_mutex_unlock(&ctx->mutex);
> + if (!pktl) {
> + if (avctx->flags & AVFMT_FLAG_NONBLOCK) {
> + return AVERROR(EAGAIN);
> + }
> + }
> + }
> +
I think this could be simplified using AVThreadMessageQueue.
> + return ctx->read_closing ? AVERROR(EOF) : pkt->size;
AVERROR_EOF
> +}
> +
> +#define OFFSET(x) offsetof(android_capture_ctx, x)
> +#define DEC AV_OPT_FLAG_DECODING_PARAM
> +static const AVOption options[] = {
> + { "video_size", "set video size given a string such as 640x480 or hd720.", OFFSET(requested_width), AV_OPT_TYPE_IMAGE_SIZE, {.str = NULL}, 0, 0, DEC },
> + { "framerate", "set video frame rate", OFFSET(framerate), AV_OPT_TYPE_STRING, {.str = NULL}, 0, 0, DEC },
> + { "sample_rate", "set audio sample rate", OFFSET(sample_rate), AV_OPT_TYPE_INT, {.i64 = 0}, 0, INT_MAX, DEC },
> + { "sample_size", "set audio sample size", OFFSET(sample_size), AV_OPT_TYPE_INT, {.i64 = 0}, 0, 16, DEC },
> + { "channels", "set number of audio channels, such as 1 or 2", OFFSET(channels), AV_OPT_TYPE_INT, {.i64 = 0}, 0, INT_MAX, DEC },
> + { "audio_buffer_size", "set audio device buffer latency size in milliseconds (default is the device's default)", OFFSET(audio_buffer_size), AV_OPT_TYPE_INT, {.i64 = 0}, 0, INT_MAX, DEC },
> + { NULL },
> +};
> +
> +static const AVClass android_capture_class = {
> + .class_name = "android_capture indev",
> + .item_name = av_default_item_name,
> + .option = options,
> + .version = LIBAVDEVICE_VERSION_INT,
> + .category = AV_CLASS_CATEGORY_DEVICE_VIDEO_INPUT,
> +};
> +
> +AVInputFormat ff_android_capture_demuxer = {
> + .name = "android_capture",
> + .long_name = NULL_IF_CONFIG_SMALL("Android camera/microphone input source via Android NDK APIs (Audio yet to be implemented)"),
> + .priv_data_size = sizeof(android_capture_ctx),
> + .read_header = android_capture_read_header,
> + .read_packet = android_capture_read_packet,
> + .read_close = android_capture_read_close,
> + .flags = AVFMT_NOFILE,
> + .priv_class = &android_capture_class,
> +};
> diff --git a/libavdevice/android_capture.h b/libavdevice/android_capture.h
> new file mode 100644
> index 0000000000..44c888ea67
> --- /dev/null
> +++ b/libavdevice/android_capture.h
> @@ -0,0 +1,77 @@
> +/*
> + * Android camera/microphone input source via Android NDK APIs (Audio yet to be implemented)
> + *
> + * Copyright (C) 2017 Felix Matouschek
> + *
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#ifndef AVDEVICE_ANDROID_CAPTURE_H
> +#define AVDEVICE_ANDROID_CAPTURE_H
> +
> +#include <pthread.h>
> +
> +#include <camera/NdkCameraManager.h>
> +#include <media/NdkImageReader.h>
> +
> +#include "libavutil/log.h"
> +#include "libavformat/avformat.h"
> +
> +typedef struct android_capture_ctx {
As much as I dislike it, the preferred form for context types is
CamelCase.
> + const AVClass *class;
> +
> + int requested_width;
> + int requested_height;
> + char *framerate;
> + int sample_rate;
> + int sample_size;
> + int channels;
> + int audio_buffer_size;
> +
> + int video_device_number;
> + int audio_device_number;
> + uint8_t lens_facing;
> + int32_t sensor_orientation;
> + int width;
> + int height;
> + AVRational requested_framerate;
> + int32_t framerate_range[2];
> + int video_stream_index;
> +
> + ACameraManager *camera_mgr;
> + char *camera_id;
> + ACameraMetadata *camera_metadata;
> + ACameraDevice *camera_dev;
> + ACameraDevice_stateCallbacks camera_state_callbacks;
> + AImageReader *image_reader;
> + AImageReader_ImageListener image_listener;
> + ANativeWindow *image_reader_window;
> + ACaptureSessionOutputContainer *capture_session_output_container;
> + ACaptureSessionOutput *capture_session_output;
> + ACameraOutputTarget *camera_output_target;
> + ACaptureRequest *capture_request;
> + ACameraCaptureSession_stateCallbacks capture_session_state_callbacks;
> + ACameraCaptureSession *capture_session;
> +
> + AVPacketList *pktl;
> + pthread_mutex_t mutex;
> + int read_closing;
> + int64_t curbufsize[2];
> + int64_t video_frame_num;
> +} android_capture_ctx;
> +
> +#endif /* AVDEVICE_ANDROID_CAPTURE_H */
Regards,
--
Nicolas George
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Digital signature
URL: <http://ffmpeg.org/pipermail/ffmpeg-devel/attachments/20171102/67219bdb/attachment.sig>
More information about the ffmpeg-devel
mailing list