[FFmpeg-devel] [PATCH] Vulkan hwcontext and filters
Mark Thompson
sw at jkqxz.net
Wed Jan 22 01:03:26 EET 2020
On 10/01/2020 21:05, Lynne wrote:
> From 04c1836f89d89dcdc892cef66ee82afbcfda9f2d Mon Sep 17 00:00:00 2001
> From: Lynne <dev at lynne.ee>
> Date: Sun, 27 Oct 2019 14:44:00 +0000
> Subject: [PATCH 4/9] lavfi: add Vulkan filtering framework
>
> This commit adds a Vulkan filtering infrastructure for libavfilter.
> It attempts to abstract as much as possible of the Vulkan API from filters.
>
> The way the hwcontext and the framework are designed permits for parallel,
> non-CPU-blocking filtering throughout, with the exception of up/downloading
> and mapping.
> ---
> configure | 3 +
> libavfilter/Makefile | 2 +
> libavfilter/glslang.cpp | 215 +++++++
> libavfilter/glslang.h | 49 ++
> libavfilter/vulkan.c | 1221 +++++++++++++++++++++++++++++++++++++++
> libavfilter/vulkan.h | 323 +++++++++++
> 6 files changed, 1813 insertions(+)
> create mode 100644 libavfilter/glslang.cpp
> create mode 100644 libavfilter/glslang.h
> create mode 100644 libavfilter/vulkan.c
> create mode 100644 libavfilter/vulkan.h
>
> diff --git a/configure b/configure
> index 3113ebfdd8..fc075f2a15 100755
> --- a/configure
> +++ b/configure
> ...
> @@ -6258,6 +6260,7 @@ enabled fontconfig && enable libfontconfig
> enabled libfontconfig && require_pkg_config libfontconfig fontconfig "fontconfig/fontconfig.h" FcInit
> enabled libfreetype && require_pkg_config libfreetype freetype2 "ft2build.h FT_FREETYPE_H" FT_Init_FreeType
> enabled libfribidi && require_pkg_config libfribidi fribidi fribidi.h fribidi_version_info
> +enabled libglslang && require_cpp libglslang glslang/SPIRV/GlslangToSpv.h "glslang::TIntermediate*" -lglslang -lOSDependent -lHLSL -lOGLCompiler -lSPVRemapper -lSPIRV -lSPIRV-Tools -lSPIRV-Tools-opt -lpthread -lstdc++
Using Debian-packaged glslang, the headers seem to be in slightly different places: SPIRV headers in their own directory under include, so it is just SPIRV/GlslangToSpv.h. With the paths edited, the version in testing works.
I don't whether there is a single official way which we should support, but this looks like it could be an evil source of confusion.
> enabled libgme && { check_pkg_config libgme libgme gme/gme.h gme_new_emu ||
> require libgme gme/gme.h gme_new_emu -lgme -lstdc++; }
> enabled libgsm && { for gsm_hdr in "gsm.h" "gsm/gsm.h"; do
> ...
> diff --git a/libavfilter/glslang.cpp b/libavfilter/glslang.cpp
> new file mode 100644
> index 0000000000..cf534d8ac5
> --- /dev/null
> +++ b/libavfilter/glslang.cpp
> @@ -0,0 +1,215 @@
> +/*
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#include <assert.h>
> +#include <pthread.h>
> +
> +extern "C" {
> +#include "libavutil/mem.h"
> +}
> +
> +#include <glslang/Include/ResourceLimits.h>
> +#include <glslang/Include/revision.h>
> +#include <glslang/Public/ShaderLang.h>
> +#include <glslang/SPIRV/GlslangToSpv.h>
(This path too.)
> +
> +#include "glslang.h"
> +
> +using namespace glslang;
> +
> +static pthread_mutex_t glslang_mutex = PTHREAD_MUTEX_INITIALIZER;
> +static int glslang_refcount;
> +
> +int glslang_init(void)
> +{
> + int ret = 1;
> +
> + pthread_mutex_lock(&glslang_mutex);
> + if (glslang_refcount++ == 0)
> + ret = InitializeProcess();
> + pthread_mutex_unlock(&glslang_mutex);
> +
> + return ret;
Inverting the normal sense of success is weird.
> +}
> +
> +void glslang_uninit(void)
> +{
> + pthread_mutex_lock(&glslang_mutex);
> + if (--glslang_refcount == 0)
> + FinalizeProcess();
> + if (glslang_refcount < 0)
Assert? This looks like a "something has gone horribly wrong" case.
> + glslang_refcount = 0;
> + pthread_mutex_unlock(&glslang_mutex);
> +}
> +
> +#define GLSL_VERSION EShTargetVulkan_1_0
> +#define SPIRV_VERSION EShTargetSpv_1_0
> +
> +extern const TBuiltInResource DefaultTBuiltInResource;
> +
> +GLSlangResult *glslang_compile(const char *glsl, enum GLSlangStage stage)
> +{
> + GLSlangResult *res = (GLSlangResult *)av_mallocz(sizeof(*res));
> +
> + static const EShLanguage lang[] = {
> + [GLSLANG_VERTEX] = EShLangVertex,
> + [GLSLANG_FRAGMENT] = EShLangFragment,
> + [GLSLANG_COMPUTE] = EShLangCompute,
> + };
> +
> + assert(glslang_refcount);
> + TShader *shader = new TShader(lang[stage]);
Missing check?
> + shader->setEnvClient(EShClientVulkan, GLSL_VERSION);
> + shader->setEnvTarget(EShTargetSpv, SPIRV_VERSION);
> + shader->setStrings(&glsl, 1);
> + if (!shader->parse(&DefaultTBuiltInResource, GLSL_VERSION, true, EShMsgDefault)) {
> + res->error_msg = av_strdup(shader->getInfoLog());
> + delete shader;
> + return res;
> + }
> +
> + TProgram *prog = new TProgram();
And here.
> + prog->addShader(shader);
> + if (!prog->link(EShMsgDefault)) {
> + res->error_msg = av_strdup(prog->getInfoLog());
> + delete shader;
> + delete prog;
> + return res;
> + }
> +
> + std::vector<unsigned int> spirv;
> + GlslangToSpv(*prog->getIntermediate(lang[stage]), spirv);
> +
> + res->success = true;
> + res->size = spirv.size() * sizeof(unsigned int);
> + res->data = av_memdup(spirv.data(), res->size),
It doesn't look like this is meant to be a comma ^
Return value check.
> + delete shader;
> + delete prog;
> + return res;
> +}
> +
> +// Taken from glslang's examples, which apparently generally bases the choices
> +// on OpenGL specification limits
> +const TBuiltInResource DefaultTBuiltInResource = {
> + /* .MaxLights = */ 32,
> + /* .MaxClipPlanes = */ 6,
> + /* .MaxTextureUnits = */ 32,
> + /* .MaxTextureCoords = */ 32,
> + /* .MaxVertexAttribs = */ 64,
> + /* .MaxVertexUniformComponents = */ 4096,
> + /* .MaxVaryingFloats = */ 64,
> + /* .MaxVertexTextureImageUnits = */ 32,
> + /* .MaxCombinedTextureImageUnits = */ 80,
> + /* .MaxTextureImageUnits = */ 32,
> + /* .MaxFragmentUniformComponents = */ 4096,
> + /* .MaxDrawBuffers = */ 32,
> + /* .MaxVertexUniformVectors = */ 128,
> + /* .MaxVaryingVectors = */ 8,
> + /* .MaxFragmentUniformVectors = */ 16,
> + /* .MaxVertexOutputVectors = */ 16,
> + /* .MaxFragmentInputVectors = */ 15,
> + /* .MinProgramTexelOffset = */ -8,
> + /* .MaxProgramTexelOffset = */ 7,
> + /* .MaxClipDistances = */ 8,
> + /* .MaxComputeWorkGroupCountX = */ 65535,
> + /* .MaxComputeWorkGroupCountY = */ 65535,
> + /* .MaxComputeWorkGroupCountZ = */ 65535,
> + /* .MaxComputeWorkGroupSizeX = */ 1024,
> + /* .MaxComputeWorkGroupSizeY = */ 1024,
> + /* .MaxComputeWorkGroupSizeZ = */ 64,
> + /* .MaxComputeUniformComponents = */ 1024,
> + /* .MaxComputeTextureImageUnits = */ 16,
> + /* .MaxComputeImageUniforms = */ 8,
> + /* .MaxComputeAtomicCounters = */ 8,
> + /* .MaxComputeAtomicCounterBuffers = */ 1,
> + /* .MaxVaryingComponents = */ 60,
> + /* .MaxVertexOutputComponents = */ 64,
> + /* .MaxGeometryInputComponents = */ 64,
> + /* .MaxGeometryOutputComponents = */ 128,
> + /* .MaxFragmentInputComponents = */ 128,
> + /* .MaxImageUnits = */ 8,
> + /* .MaxCombinedImageUnitsAndFragmentOutputs = */ 8,
> + /* .MaxCombinedShaderOutputResources = */ 8,
> + /* .MaxImageSamples = */ 0,
> + /* .MaxVertexImageUniforms = */ 0,
> + /* .MaxTessControlImageUniforms = */ 0,
> + /* .MaxTessEvaluationImageUniforms = */ 0,
> + /* .MaxGeometryImageUniforms = */ 0,
> + /* .MaxFragmentImageUniforms = */ 8,
> + /* .MaxCombinedImageUniforms = */ 8,
> + /* .MaxGeometryTextureImageUnits = */ 16,
> + /* .MaxGeometryOutputVertices = */ 256,
> + /* .MaxGeometryTotalOutputComponents = */ 1024,
> + /* .MaxGeometryUniformComponents = */ 1024,
> + /* .MaxGeometryVaryingComponents = */ 64,
> + /* .MaxTessControlInputComponents = */ 128,
> + /* .MaxTessControlOutputComponents = */ 128,
> + /* .MaxTessControlTextureImageUnits = */ 16,
> + /* .MaxTessControlUniformComponents = */ 1024,
> + /* .MaxTessControlTotalOutputComponents = */ 4096,
> + /* .MaxTessEvaluationInputComponents = */ 128,
> + /* .MaxTessEvaluationOutputComponents = */ 128,
> + /* .MaxTessEvaluationTextureImageUnits = */ 16,
> + /* .MaxTessEvaluationUniformComponents = */ 1024,
> + /* .MaxTessPatchComponents = */ 120,
> + /* .MaxPatchVertices = */ 32,
> + /* .MaxTessGenLevel = */ 64,
> + /* .MaxViewports = */ 16,
> + /* .MaxVertexAtomicCounters = */ 0,
> + /* .MaxTessControlAtomicCounters = */ 0,
> + /* .MaxTessEvaluationAtomicCounters = */ 0,
> + /* .MaxGeometryAtomicCounters = */ 0,
> + /* .MaxFragmentAtomicCounters = */ 8,
> + /* .MaxCombinedAtomicCounters = */ 8,
> + /* .MaxAtomicCounterBindings = */ 1,
> + /* .MaxVertexAtomicCounterBuffers = */ 0,
> + /* .MaxTessControlAtomicCounterBuffers = */ 0,
> + /* .MaxTessEvaluationAtomicCounterBuffers = */ 0,
> + /* .MaxGeometryAtomicCounterBuffers = */ 0,
> + /* .MaxFragmentAtomicCounterBuffers = */ 1,
> + /* .MaxCombinedAtomicCounterBuffers = */ 1,
> + /* .MaxAtomicCounterBufferSize = */ 16384,
> + /* .MaxTransformFeedbackBuffers = */ 4,
> + /* .MaxTransformFeedbackInterleavedComponents = */ 64,
> + /* .MaxCullDistances = */ 8,
> + /* .MaxCombinedClipAndCullDistances = */ 8,
> + /* .MaxSamples = */ 4,
> +#if GLSLANG_PATCH_LEVEL >= 2892
> + /* .maxMeshOutputVerticesNV = */ 256,
> + /* .maxMeshOutputPrimitivesNV = */ 512,
> + /* .maxMeshWorkGroupSizeX_NV = */ 32,
> + /* .maxMeshWorkGroupSizeY_NV = */ 1,
> + /* .maxMeshWorkGroupSizeZ_NV = */ 1,
> + /* .maxTaskWorkGroupSizeX_NV = */ 32,
> + /* .maxTaskWorkGroupSizeY_NV = */ 1,
> + /* .maxTaskWorkGroupSizeZ_NV = */ 1,
> + /* .maxMeshViewCountNV = */ 4,
> +#endif
> +
> + .limits = {
> + /* .nonInductiveForLoops = */ 1,
> + /* .whileLoops = */ 1,
> + /* .doWhileLoops = */ 1,
> + /* .generalUniformIndexing = */ 1,
> + /* .generalAttributeMatrixVectorIndexing = */ 1,
> + /* .generalVaryingIndexing = */ 1,
> + /* .generalSamplerIndexing = */ 1,
> + /* .generalVariableIndexing = */ 1,
> + /* .generalConstantMatrixVectorIndexing = */ 1,
> + }
Why are the designators commented out? Letting the compiler check your ordering seems like a good idea.
> +};
> diff --git a/libavfilter/glslang.h b/libavfilter/glslang.h
> new file mode 100644
> index 0000000000..865af71580
> --- /dev/null
> +++ b/libavfilter/glslang.h
> @@ -0,0 +1,49 @@
> +/*
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#pragma once
The pragma isn't used anywhere else in the codebase.
> +
> +#include <stdlib.h>
> +
> +#ifdef __cplusplus
> +extern "C" {
> +#endif
> +
> +int glslang_init(void);
> +void glslang_uninit(void);
> +
> +typedef struct GLSlangResult {
> + int success;
> + const char *error_msg;
> +
> + void *data; /* Shader data or NULL */
> + size_t size;
> +} GLSlangResult;
> +
> +enum GLSlangStage {
> + GLSLANG_VERTEX,
> + GLSLANG_FRAGMENT,
> + GLSLANG_COMPUTE,
> +};
> +
> +/* Compile GLSL into a SPIRV stream, if possible */
> +GLSlangResult *glslang_compile(const char *glsl, enum GLSlangStage stage);
+1, best API ever. (Now why can't glslang or similar projects just give us this function themselves?)
> +
> +#ifdef __cplusplus
> +}
> +#endif
> diff --git a/libavfilter/vulkan.c b/libavfilter/vulkan.c
> new file mode 100644
> index 0000000000..99aaeb2ef4
> --- /dev/null
> +++ b/libavfilter/vulkan.c
> ...
> +
> +int ff_vk_filter_config_output_inplace(AVFilterLink *outlink)
> +{
> + int err;
> + AVFilterContext *avctx = outlink->src;
> + VulkanFilterContext *s = avctx->priv;
> +
> + av_buffer_unref(&outlink->hw_frames_ctx);
> +
> + if (!s->device_ref) {
> + if (!avctx->hw_device_ctx) {
> + av_log(avctx, AV_LOG_ERROR, "Vulkan filtering requires a "
> + "Vulkan device.\n");
> + return AVERROR(EINVAL);
> + }
> +
> + err = vulkan_filter_set_device(avctx, avctx->hw_device_ctx);
> + if (err < 0)
> + return err;
> + }
> +
> + outlink->hw_frames_ctx = av_buffer_ref(s->frames_ref);
Missing check.
> + outlink->w = s->output_width;
> + outlink->h = s->output_height;
> +
> + return 0;
> +}
> ...
The rest of this all looks generally sensible, though I don't have the detailed Vulkan knowledge to go through it properly. Probably fine in any case, since it's all internal and we aren't locking anything down.
> diff --git a/libavfilter/vulkan.h b/libavfilter/vulkan.h
> new file mode 100644
> index 0000000000..8d4def1a00
> --- /dev/null
> +++ b/libavfilter/vulkan.h
> @@ -0,0 +1,323 @@
> +/*
> + * This file is part of FFmpeg.
> + *
> + * FFmpeg is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * FFmpeg is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with FFmpeg; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
> + */
> +
> +#ifndef AVFILTER_VULKAN_COMMON_H
> +#define AVFILTER_VULKAN_COMMON_H
Looks like you've renamed it since then.
> +
> ...
Thanks,
- Mark
More information about the ffmpeg-devel
mailing list