[FFmpeg-devel] [PATCH] hwcontext_vulkan: workaround MoltenVK's bug which leads to segmentation fault

["zhilizhao(赵志立)"]

> On Jan 27, 2022, at 4:18 PM, Wu, Jianhua <jianhua.wu-at-intel.com at ffmpeg.org> wrote:
> 
> Zhao Zhili wrote:
>> Sent: Thursday, January 27, 2022 4:11 PM
>> To: ffmpeg-devel at ffmpeg.org
>> Cc: Zhao Zhili <quinkblack at foxmail.com>
>> Subject: [FFmpeg-devel] [PATCH] hwcontext_vulkan: workaround
>> MoltenVK's bug which leads to segmentation fault
>> 
>> MoltenVK doesn't reset instance pointer when CreateInstance() failed, then
>> DestroyInstance() leads to segmentation fault. MoltenVK's bug has been
>> fixed by [1], which doesn't available on homebrew yet.
>> Regardless MoltenVK's bug, we shouldn't call DestroyInstance() in the case
>> of CreateInstance() failed, so reset instance making sense.
>> 
>> [1] https://github.com/KhronosGroup/MoltenVK/commit/86a1fbdb8
>> ---
>> libavutil/hwcontext_vulkan.c | 2 ++
>> 1 file changed, 2 insertions(+)
>> 
>> diff --git a/libavutil/hwcontext_vulkan.c b/libavutil/hwcontext_vulkan.c
>> index 2e219511c9..ac8e00003a 100644
>> --- a/libavutil/hwcontext_vulkan.c
>> +++ b/libavutil/hwcontext_vulkan.c
>> @@ -719,6 +719,8 @@ static int create_instance(AVHWDeviceContext *ctx,
>> AVDictionary *opts)
>>     if (ret != VK_SUCCESS) {
>>         av_log(ctx, AV_LOG_ERROR, "Instance creation failure: %s\n",
>>                vk_ret2str(ret));
>> +        /* Workaround MoltenVK's bug which doesn't reset instance pointer.
>> */
>> +        hwctx->inst = (VkInstance) { 0 };
> 
> Hi,
> 
> It's no need to use the explicit cast and use hwctx->inst = VK_NULL_HANDLE instead, which is the null context defined by Vulkan spec.

I didn’t notice that, thanks! Patch v2:

http://ffmpeg.org/pipermail/ffmpeg-devel/2022-January/292181.html

> 
> Thanks,
> Jianhua
> 
> _______________________________________________
> ffmpeg-devel mailing list
> ffmpeg-devel at ffmpeg.org
> https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
> 
> To unsubscribe, visit link above, or email
> ffmpeg-devel-request at ffmpeg.org with subject "unsubscribe".