[FFmpeg-devel] [PATCH v2] avfilter/vf_edgedetect: properly implement double_threshold()

[Valery Kot]

=== Version 1 ===
vf_edgedetect video filter implements Canny algorithm
(https://en.wikipedia.org/wiki/Canny_edge_detector)

Important part of this algo is the double threshold step: pixels above
"high" threshold being kept, pixels below "low" threshold dropped,
pixels in between kept if they are attached to "high" pixels.

This is implemented in the double_threshold() function. However,
condition to start checking attached pixels, as it is now and as it
was in FFmpeg since 2012, only allows checking on the boundary, not
inside the video. It is a very lucky coincidence that those boundary
pixels are always 0, otherwise following lines would be reading
outside of the buffer.

As it is now, double_threshold() only implements "high" thresholding.
As a result, edges are either noisy or fragmented, depending on "high"
threshold selection; "low" threshold is simply ignored.

Attached one char patch fixes this.

=== Version 2 ===
- include avfilter/ in commit message
- update FATE tests
-------------- next part --------------
From 69bbe24bfe23efa3874448f28451b1abaa209d5d Mon Sep 17 00:00:00 2001
From: vkot <valery.kot at kinetiq.tv>
Date: Fri, 19 Jun 2020 16:57:13 +0200
Subject: [PATCH] avfilter/vf_edgedetect: properly implement double_threshold()

---
 libavfilter/vf_edgedetect.c               | 2 +-
 tests/ref/fate/filter-edgedetect          | 2 +-
 tests/ref/fate/filter-edgedetect-colormix | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/libavfilter/vf_edgedetect.c b/libavfilter/vf_edgedetect.c
index a5614ea63b..df8afbd532 100644
--- a/libavfilter/vf_edgedetect.c
+++ b/libavfilter/vf_edgedetect.c
@@ -294,7 +294,7 @@ static void double_threshold(int low, int high, int w, int h,
                 continue;
             }
 
-            if ((!i || i == w - 1 || !j || j == h - 1) &&
+            if (!(!i || i == w - 1 || !j || j == h - 1) &&
                 src[i] > low &&
                 (src[-src_linesize + i-1] > high ||
                  src[-src_linesize + i  ] > high ||
diff --git a/tests/ref/fate/filter-edgedetect b/tests/ref/fate/filter-edgedetect
index 23c9953e61..e49639afac 100644
--- a/tests/ref/fate/filter-edgedetect
+++ b/tests/ref/fate/filter-edgedetect
@@ -1 +1 @@
-edgedetect          93ceace33f6636bcdbeb037317c65745
+edgedetect          04ff46bb35edff3dbad4102391516d25
diff --git a/tests/ref/fate/filter-edgedetect-colormix b/tests/ref/fate/filter-edgedetect-colormix
index e828c6bd19..0df17344bc 100644
--- a/tests/ref/fate/filter-edgedetect-colormix
+++ b/tests/ref/fate/filter-edgedetect-colormix
@@ -1 +1 @@
-edgedetect-colormix 1b8658252e2f03fbae30e6d63dd24c7c
+edgedetect-colormix 9f50c5586f899a8f5a10059154d64bde
-- 
2.26.2.windows.1