[FFmpeg-devel] [PATCH] Vulkan hwcontext and filters

Lynne dev at lynne.ee
Sat Feb 1 14:19:34 EET 2020


Jan 21, 2020, 23:03 by sw at jkqxz.net:

> On 10/01/2020 21:05, Lynne wrote:
>
>> From 04c1836f89d89dcdc892cef66ee82afbcfda9f2d Mon Sep 17 00:00:00 2001
>> From: Lynne <dev at lynne.ee>
>> Date: Sun, 27 Oct 2019 14:44:00 +0000
>> Subject: [PATCH 4/9] lavfi: add Vulkan filtering framework
>>
>> This commit adds a Vulkan filtering infrastructure for libavfilter.
>> It attempts to abstract as much as possible of the Vulkan API from filters.
>>
>> The way the hwcontext and the framework are designed permits for parallel,
>> non-CPU-blocking filtering throughout, with the exception of up/downloading
>> and mapping.
>> ---
>>  configure               |    3 +
>>  libavfilter/Makefile    |    2 +
>>  libavfilter/glslang.cpp |  215 +++++++
>>  libavfilter/glslang.h   |   49 ++
>>  libavfilter/vulkan.c    | 1221 +++++++++++++++++++++++++++++++++++++++
>>  libavfilter/vulkan.h    |  323 +++++++++++
>>  6 files changed, 1813 insertions(+)
>>  create mode 100644 libavfilter/glslang.cpp
>>  create mode 100644 libavfilter/glslang.h
>>  create mode 100644 libavfilter/vulkan.c
>>  create mode 100644 libavfilter/vulkan.h
>>
>> diff --git a/configure b/configure
>> index 3113ebfdd8..fc075f2a15 100755
>> --- a/configure
>> +++ b/configure
>> ...
>> @@ -6258,6 +6260,7 @@ enabled fontconfig        && enable libfontconfig
>>  enabled libfontconfig     && require_pkg_config libfontconfig fontconfig "fontconfig/fontconfig.h" FcInit
>>  enabled libfreetype       && require_pkg_config libfreetype freetype2 "ft2build.h FT_FREETYPE_H" FT_Init_FreeType
>>  enabled libfribidi        && require_pkg_config libfribidi fribidi fribidi.h fribidi_version_info
>> +enabled libglslang        && require_cpp libglslang glslang/SPIRV/GlslangToSpv.h "glslang::TIntermediate*" -lglslang -lOSDependent -lHLSL -lOGLCompiler -lSPVRemapper -lSPIRV -lSPIRV-Tools -lSPIRV-Tools-opt -lpthread -lstdc++
>>
>
> Using Debian-packaged glslang, the headers seem to be in slightly different places: SPIRV headers in their own directory under include, so it is just SPIRV/GlslangToSpv.h.  With the paths edited, the version in testing works.
>
> I don't whether there is a single official way which we should support, but this looks like it could be an evil source of confusion.
>

glslang change the headers directory from just SPIRV to glslang/SPIRV. Debian's package already contains this, and so does arch, and I'd rather not have ifdeffery and additional header checks, so I'll keep it like this.



>> enabled libgme            && { check_pkg_config libgme libgme gme/gme.h gme_new_emu ||
>>  require libgme gme/gme.h gme_new_emu -lgme -lstdc++; }
>>  enabled libgsm            && { for gsm_hdr in "gsm.h" "gsm/gsm.h"; do
>> ...
>> diff --git a/libavfilter/glslang.cpp b/libavfilter/glslang.cpp
>> new file mode 100644
>> index 0000000000..cf534d8ac5
>> --- /dev/null
>> +++ b/libavfilter/glslang.cpp
>> @@ -0,0 +1,215 @@
>> +/*
>> + * This file is part of FFmpeg.
>> + *
>> + * FFmpeg is free software; you can redistribute it and/or
>> + * modify it under the terms of the GNU Lesser General Public
>> + * License as published by the Free Software Foundation; either
>> + * version 2.1 of the License, or (at your option) any later version.
>> + *
>> + * FFmpeg is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>> + * Lesser General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU Lesser General Public
>> + * License along with FFmpeg; if not, write to the Free Software
>> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
>> + */
>> +
>> +#include <assert.h>
>> +#include <pthread.h>
>> +
>> +extern "C" {
>> +#include "libavutil/mem.h"
>> +}
>> +
>> +#include <glslang/Include/ResourceLimits.h>
>> +#include <glslang/Include/revision.h>
>> +#include <glslang/Public/ShaderLang.h>
>> +#include <glslang/SPIRV/GlslangToSpv.h>
>>
>
> (This path too.)
>
>> +
>> +#include "glslang.h"
>> +
>> +using namespace glslang;
>> +
>> +static pthread_mutex_t glslang_mutex = PTHREAD_MUTEX_INITIALIZER;
>> +static int glslang_refcount;
>> +
>> +int glslang_init(void)
>> +{
>> +    int ret = 1;
>> +
>> +    pthread_mutex_lock(&glslang_mutex);
>> +    if (glslang_refcount++ == 0)
>> +        ret = InitializeProcess();
>> +    pthread_mutex_unlock(&glslang_mutex);
>> +
>> +    return ret;
>>
>
> Inverting the normal sense of success is weird.
>

Fixed.



>> +}
>> +
>> +void glslang_uninit(void)
>> +{
>> +    pthread_mutex_lock(&glslang_mutex);
>> +    if (--glslang_refcount == 0)
>> +        FinalizeProcess();
>> +    if (glslang_refcount < 0)
>>
>
> Assert?  This looks like a "something has gone horribly wrong" case.
>

Fixed.



>> +        glslang_refcount = 0;
>> +    pthread_mutex_unlock(&glslang_mutex);
>> +}
>> +
>> +#define GLSL_VERSION EShTargetVulkan_1_0
>> +#define SPIRV_VERSION EShTargetSpv_1_0
>> +
>> +extern const TBuiltInResource DefaultTBuiltInResource;
>> +
>> +GLSlangResult *glslang_compile(const char *glsl, enum GLSlangStage stage)
>> +{
>> +    GLSlangResult *res = (GLSlangResult *)av_mallocz(sizeof(*res));
>> +
>> +    static const EShLanguage lang[] = {
>> +        [GLSLANG_VERTEX]   = EShLangVertex,
>> +        [GLSLANG_FRAGMENT] = EShLangFragment,
>> +        [GLSLANG_COMPUTE]  = EShLangCompute,
>> +    };
>> +
>> +    assert(glslang_refcount);
>> +    TShader *shader = new TShader(lang[stage]);
>>
>
> Missing check?
>

Fixed all.



>> +    shader->setEnvClient(EShClientVulkan, GLSL_VERSION);
>> +    shader->setEnvTarget(EShTargetSpv, SPIRV_VERSION);
>> +    shader->setStrings(&glsl, 1);
>> +    if (!shader->parse(&DefaultTBuiltInResource, GLSL_VERSION, true, EShMsgDefault)) {
>> +        res->error_msg = av_strdup(shader->getInfoLog());
>> +        delete shader;
>> +        return res;
>> +    }
>> +
>> +    TProgram *prog = new TProgram();
>>
>
> And here.
>
>> +    prog->addShader(shader);
>> +    if (!prog->link(EShMsgDefault)) {
>> +        res->error_msg = av_strdup(prog->getInfoLog());
>> +        delete shader;
>> +        delete prog;
>> +        return res;
>> +    }
>> +
>> +    std::vector<unsigned int> spirv;
>> +    GlslangToSpv(*prog->getIntermediate(lang[stage]), spirv);
>> +
>> +    res->success = true;
>> +    res->size = spirv.size() * sizeof(unsigned int);
>> +    res->data = av_memdup(spirv.data(), res->size),
>>
>
> It doesn't look like this is meant to be a comma     ^
>
> Return value check.
>
>> +    delete shader;
>> +    delete prog;
>> +    return res;
>> +}
>> +
>> +// Taken from glslang's examples, which apparently generally bases the choices
>> +// on OpenGL specification limits
>> +const TBuiltInResource DefaultTBuiltInResource = {
>> +    /* .MaxLights = */ 32,
>> +    /* .MaxClipPlanes = */ 6,
>> +    /* .MaxTextureUnits = */ 32,
>> +    /* .MaxTextureCoords = */ 32,
>> +    /* .MaxVertexAttribs = */ 64,
>> +    /* .MaxVertexUniformComponents = */ 4096,
>> +    /* .MaxVaryingFloats = */ 64,
>> +    /* .MaxVertexTextureImageUnits = */ 32,
>> +    /* .MaxCombinedTextureImageUnits = */ 80,
>> +    /* .MaxTextureImageUnits = */ 32,
>> +    /* .MaxFragmentUniformComponents = */ 4096,
>> +    /* .MaxDrawBuffers = */ 32,
>> +    /* .MaxVertexUniformVectors = */ 128,
>> +    /* .MaxVaryingVectors = */ 8,
>> +    /* .MaxFragmentUniformVectors = */ 16,
>> +    /* .MaxVertexOutputVectors = */ 16,
>> +    /* .MaxFragmentInputVectors = */ 15,
>> +    /* .MinProgramTexelOffset = */ -8,
>> +    /* .MaxProgramTexelOffset = */ 7,
>> +    /* .MaxClipDistances = */ 8,
>> +    /* .MaxComputeWorkGroupCountX = */ 65535,
>> +    /* .MaxComputeWorkGroupCountY = */ 65535,
>> +    /* .MaxComputeWorkGroupCountZ = */ 65535,
>> +    /* .MaxComputeWorkGroupSizeX = */ 1024,
>> +    /* .MaxComputeWorkGroupSizeY = */ 1024,
>> +    /* .MaxComputeWorkGroupSizeZ = */ 64,
>> +    /* .MaxComputeUniformComponents = */ 1024,
>> +    /* .MaxComputeTextureImageUnits = */ 16,
>> +    /* .MaxComputeImageUniforms = */ 8,
>> +    /* .MaxComputeAtomicCounters = */ 8,
>> +    /* .MaxComputeAtomicCounterBuffers = */ 1,
>> +    /* .MaxVaryingComponents = */ 60,
>> +    /* .MaxVertexOutputComponents = */ 64,
>> +    /* .MaxGeometryInputComponents = */ 64,
>> +    /* .MaxGeometryOutputComponents = */ 128,
>> +    /* .MaxFragmentInputComponents = */ 128,
>> +    /* .MaxImageUnits = */ 8,
>> +    /* .MaxCombinedImageUnitsAndFragmentOutputs = */ 8,
>> +    /* .MaxCombinedShaderOutputResources = */ 8,
>> +    /* .MaxImageSamples = */ 0,
>> +    /* .MaxVertexImageUniforms = */ 0,
>> +    /* .MaxTessControlImageUniforms = */ 0,
>> +    /* .MaxTessEvaluationImageUniforms = */ 0,
>> +    /* .MaxGeometryImageUniforms = */ 0,
>> +    /* .MaxFragmentImageUniforms = */ 8,
>> +    /* .MaxCombinedImageUniforms = */ 8,
>> +    /* .MaxGeometryTextureImageUnits = */ 16,
>> +    /* .MaxGeometryOutputVertices = */ 256,
>> +    /* .MaxGeometryTotalOutputComponents = */ 1024,
>> +    /* .MaxGeometryUniformComponents = */ 1024,
>> +    /* .MaxGeometryVaryingComponents = */ 64,
>> +    /* .MaxTessControlInputComponents = */ 128,
>> +    /* .MaxTessControlOutputComponents = */ 128,
>> +    /* .MaxTessControlTextureImageUnits = */ 16,
>> +    /* .MaxTessControlUniformComponents = */ 1024,
>> +    /* .MaxTessControlTotalOutputComponents = */ 4096,
>> +    /* .MaxTessEvaluationInputComponents = */ 128,
>> +    /* .MaxTessEvaluationOutputComponents = */ 128,
>> +    /* .MaxTessEvaluationTextureImageUnits = */ 16,
>> +    /* .MaxTessEvaluationUniformComponents = */ 1024,
>> +    /* .MaxTessPatchComponents = */ 120,
>> +    /* .MaxPatchVertices = */ 32,
>> +    /* .MaxTessGenLevel = */ 64,
>> +    /* .MaxViewports = */ 16,
>> +    /* .MaxVertexAtomicCounters = */ 0,
>> +    /* .MaxTessControlAtomicCounters = */ 0,
>> +    /* .MaxTessEvaluationAtomicCounters = */ 0,
>> +    /* .MaxGeometryAtomicCounters = */ 0,
>> +    /* .MaxFragmentAtomicCounters = */ 8,
>> +    /* .MaxCombinedAtomicCounters = */ 8,
>> +    /* .MaxAtomicCounterBindings = */ 1,
>> +    /* .MaxVertexAtomicCounterBuffers = */ 0,
>> +    /* .MaxTessControlAtomicCounterBuffers = */ 0,
>> +    /* .MaxTessEvaluationAtomicCounterBuffers = */ 0,
>> +    /* .MaxGeometryAtomicCounterBuffers = */ 0,
>> +    /* .MaxFragmentAtomicCounterBuffers = */ 1,
>> +    /* .MaxCombinedAtomicCounterBuffers = */ 1,
>> +    /* .MaxAtomicCounterBufferSize = */ 16384,
>> +    /* .MaxTransformFeedbackBuffers = */ 4,
>> +    /* .MaxTransformFeedbackInterleavedComponents = */ 64,
>> +    /* .MaxCullDistances = */ 8,
>> +    /* .MaxCombinedClipAndCullDistances = */ 8,
>> +    /* .MaxSamples = */ 4,
>> +#if GLSLANG_PATCH_LEVEL >= 2892
>> +    /* .maxMeshOutputVerticesNV = */ 256,
>> +    /* .maxMeshOutputPrimitivesNV = */ 512,
>> +    /* .maxMeshWorkGroupSizeX_NV = */ 32,
>> +    /* .maxMeshWorkGroupSizeY_NV = */ 1,
>> +    /* .maxMeshWorkGroupSizeZ_NV = */ 1,
>> +    /* .maxTaskWorkGroupSizeX_NV = */ 32,
>> +    /* .maxTaskWorkGroupSizeY_NV = */ 1,
>> +    /* .maxTaskWorkGroupSizeZ_NV = */ 1,
>> +    /* .maxMeshViewCountNV = */ 4,
>> +#endif
>> +
>> +    .limits = {
>> +        /* .nonInductiveForLoops = */ 1,
>> +        /* .whileLoops = */ 1,
>> +        /* .doWhileLoops = */ 1,
>> +        /* .generalUniformIndexing = */ 1,
>> +        /* .generalAttributeMatrixVectorIndexing = */ 1,
>> +        /* .generalVaryingIndexing = */ 1,
>> +        /* .generalSamplerIndexing = */ 1,
>> +        /* .generalVariableIndexing = */ 1,
>> +        /* .generalConstantMatrixVectorIndexing = */ 1,
>> +    }
>>
>
> Why are the designators commented out?  Letting the compiler check your ordering seems like a good idea.
>

glslang's nice API doesn't allow this:
error: ‘const TBuiltInResource’ has no non-static data member named ‘MaxLights’



>> +};
>> diff --git a/libavfilter/glslang.h b/libavfilter/glslang.h
>> new file mode 100644
>> index 0000000000..865af71580
>> --- /dev/null
>> +++ b/libavfilter/glslang.h
>> @@ -0,0 +1,49 @@
>> +/*
>> + * This file is part of FFmpeg.
>> + *
>> + * FFmpeg is free software; you can redistribute it and/or
>> + * modify it under the terms of the GNU Lesser General Public
>> + * License as published by the Free Software Foundation; either
>> + * version 2.1 of the License, or (at your option) any later version.
>> + *
>> + * FFmpeg is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>> + * Lesser General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU Lesser General Public
>> + * License along with FFmpeg; if not, write to the Free Software
>> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
>> + */
>> +
>> +#pragma once
>>
>
> The pragma isn't used anywhere else in the codebase.
>

Fixed.



>> +
>> +#include <stdlib.h>
>> +
>> +#ifdef __cplusplus
>> +extern "C" {
>> +#endif
>> +
>> +int glslang_init(void);
>> +void glslang_uninit(void);
>> +
>> +typedef struct GLSlangResult {
>> +    int success;
>> +    const char *error_msg;
>> +
>> +    void *data; /* Shader data or NULL */
>> +    size_t size;
>> +} GLSlangResult;
>> +
>> +enum GLSlangStage {
>> +    GLSLANG_VERTEX,
>> +    GLSLANG_FRAGMENT,
>> +    GLSLANG_COMPUTE,
>> +};
>> +
>> +/* Compile GLSL into a SPIRV stream, if possible */
>> +GLSlangResult *glslang_compile(const char *glsl, enum GLSlangStage stage);
>>
>
> +1, best API ever.  (Now why can't glslang or similar projects just give us this function themselves?)
>
>> +
>> +#ifdef __cplusplus
>> +}
>> +#endif
>> diff --git a/libavfilter/vulkan.c b/libavfilter/vulkan.c
>> new file mode 100644
>> index 0000000000..99aaeb2ef4
>> --- /dev/null
>> +++ b/libavfilter/vulkan.c
>> ...
>> +
>> +int ff_vk_filter_config_output_inplace(AVFilterLink *outlink)
>> +{
>> +    int err;
>> +    AVFilterContext *avctx = outlink->src;
>> +    VulkanFilterContext *s = avctx->priv;
>> +
>> +    av_buffer_unref(&outlink->hw_frames_ctx);
>> +
>> +    if (!s->device_ref) {
>> +        if (!avctx->hw_device_ctx) {
>> +            av_log(avctx, AV_LOG_ERROR, "Vulkan filtering requires a "
>> +                   "Vulkan device.\n");
>> +            return AVERROR(EINVAL);
>> +        }
>> +
>> +        err = vulkan_filter_set_device(avctx, avctx->hw_device_ctx);
>> +        if (err < 0)
>> +            return err;
>> +    }
>> +
>> +    outlink->hw_frames_ctx = av_buffer_ref(s->frames_ref);
>>
>
> Missing check.
>
>> +    outlink->w = s->output_width;
>> +    outlink->h = s->output_height;
>> +
>> +    return 0;
>> +}
>> ...
>>
>
> The rest of this all looks generally sensible, though I don't have the detailed Vulkan knowledge to go through it properly.  Probably fine in any case, since it's all internal and we aren't locking anything down.
>
>> diff --git a/libavfilter/vulkan.h b/libavfilter/vulkan.h
>> new file mode 100644
>> index 0000000000..8d4def1a00
>> --- /dev/null
>> +++ b/libavfilter/vulkan.h
>> @@ -0,0 +1,323 @@
>> +/*
>> + * This file is part of FFmpeg.
>> + *
>> + * FFmpeg is free software; you can redistribute it and/or
>> + * modify it under the terms of the GNU Lesser General Public
>> + * License as published by the Free Software Foundation; either
>> + * version 2.1 of the License, or (at your option) any later version.
>> + *
>> + * FFmpeg is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
>> + * Lesser General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU Lesser General Public
>> + * License along with FFmpeg; if not, write to the Free Software
>> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
>> + */
>> +
>> +#ifndef AVFILTER_VULKAN_COMMON_H
>> +#define AVFILTER_VULKAN_COMMON_H
>>
>
> Looks like you've renamed it since then.
>
>> +
>> ...
>>
>
> Thanks,
>
> - Mark
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
>
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org>  with subject "unsubscribe".
>

Updated code is in the git repo.



More information about the ffmpeg-devel mailing list