Overview of Image Edge Detection

doi:10.3778/j.issn.1002-8331.2209-0122

Abstract

Abstract: The task of edge detection is to identify pixels with significant brightness changes as target edges, which is a low-level problem in computer vision, and edge detection has important applications in object recognition and detection, object proposal generation, and image segmentation. Nowadays, edge detection has produced several types of methods, such as traditional gradient-based detection methods and deep learning-based edge detection algorithms and detection methods combined with emerging technologies. A finer classification of these methods provides researchers with a clearer understanding of the trends in edge detection. Firstly, the theoretical basis and implementation methods of traditional edge detection are introduced; then the main edge detection methods in recent years are summarized and classified according to the methods used, and the core techniques used in them are introduced, such as branching structure, feature fusion and loss function. The evaluation indicators used to assess the algorithm’s performance are single-image optimal threshold（ODS） and frame per second（FPS）, which are contrasted using the fundamental data set（BSDS500）. Finally, the current state of edge detection research is examined and summarized, and the possible future research directions of edge detection are prospected.

Key words: edge detection, gradient operator, deep learning, feature fusion, loss function

摘要： 边缘检测的任务是将亮度变化明显的像素点识别为目标边缘，是计算机视觉低层级问题，并且边缘检测在对象识别和检测、对象提议生成、图像分割有着重要应用。如今，边缘检测已经产生了多类方法，如基于梯度的传统检测方法、基于深度学习的边缘检测算法，还有结合新兴技术的检测方法等。对这些方法进行更精细的分类，让研究者更清楚地了解边缘检测的发展趋势。对传统边缘检测的理论依据及实现方法做出介绍；详细介绍近年来主要的深度学习边缘检测方法，根据使用的方法进行分类，并对其中所使用的创新技术进行说明，如分支结构、特征融合和损失函数。衡量算法性能采用评估指标：单图最佳阈值（ODS）和帧数（FPS），在基础数据集（BSDS500）上进行对比。对边缘检测的研究现状进行分析和总结，对未来可能的研究方向进行展望。

关键词: 边缘检测, 梯度算子, 深度学习, 特征融合, 损失函数

XIAO Yang, ZHOU Jun. Overview of Image Edge Detection[J]. Computer Engineering and Applications, 2023, 59(5): 40-54.

肖扬, 周军. 图像边缘检测综述[J]. 计算机工程与应用, 2023, 59(5): 40-54.

References

[1] 李翠锦，瞿中.基于深度学习的图像边缘检测算法综述[J].计算机应用，2020，40（11）：3280-3288.
LI C J，QU Z.Review of image edge detection algorithms based on deep learning[J].Journal of Computer Applications，2020，40（11）：3280-3288.
[2] KITTLER J.On the accuracy of the Sobel edge detector[J].Image and Vision Computing，1983，1（1）：37-42.
[3] ROBERTS L G.Machine perception of three-dimensional solids[D].Massachusetts Institute of Technology，1963.
[4] LECUN Y，BOTTOU L，BENGIO Y，et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE，1998，86（11）：2278-2324.
[5] CANNY J.A computational approach to edge detection[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，1986（6）：679-698.
[6] KONISHI S，YUILLE A L，COUGHLAN J M，et al.Statistical edge detection：learning and evaluating edge cues[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2003，25（1）：57-74.
[7] MARTIN D R，FOWLKES C C，MALIK J.Learning to detect natural image boundaries using local brightness，color，and texture cues[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2004，26（5）：530-549.
[8] ISOLA P，ZORAN D，KRISHNAN D，et al.Crisp boundary detection using pointwise mutual information[C]//European Conference on Computer Vision.Cham：Springer，2014：799-814.
[9] DOLLáR P，ZITNICK C L.Fast edge detection using structured forests[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2014，37（8）：1558-1570.
[10] ARBELAEZ P，MAIRE M，FOWLKES C，et al.Contour detection and hierarchical image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2010，33（5）：898-916.
[11] GUPTA S，ARBELAEZ P，MALIK J.Perceptual organization and recognition of indoor scenes from RGB-D images[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2013：564-571.
[12] EVERINGHAM M，ESLAMI S M，VAN GOOL L，et al.The pascal visual object classes challenge：a retrospective[J].International Journal of Computer Vision，2015，111（1）：98-136.
[13] XIE S，TU Z.Holistically-nested edge detection[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：1395-1403.
[14] SU Z，LIU W，YU Z，et al.Pixel difference networks for efficient edge detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：5117-5127.
[15] LIU Y，LEW M S.Learning relaxed deep supervision for better edge detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：231-240.
[16] WANG Y，ZHAO X，LI Y，et al.Deep crisp boundaries：from boundaries to higher-level tasks[J].IEEE Transactions on Image Processing，2018，28（3）：1285-1298.
[17] DENG R，LIU S.Deep structural contour detection[C]//Proceedings of the 28th ACM International Conference on Multimedia，2020：304-312.
[18] YUAN G X，HO C H，LIN C J.Recent advances of large-scale linear classification[J].Proceedings of the IEEE，2012，100（9）：2584-2603.
[19] CORTES C，VAPNIK V.Support-vector networks[J].Machine Learning，1995，20（3）：273-297.
[20] PREWITT J M S.Object enhancement and extraction[J].Picture Processing and Psychopictorics，1970，10（1）：15-19.
[21] KIRSCH R A.Computer determination of the constituent structure of biological images[J].Computers and Biomedical Research，1971，4（3）：315-328.
[22] TORRE V，POGGIO T A.On edge detection[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，1986（2）：147-163.
[23] LOWE D G.Object recognition from local scale-invariant features[C]//Proceedings of the Seventh IEEE International Conference on Computer Vision，1999：1150-1157.
[24] COVER T M.Elements of information theory[M].[S.l.]：John Wiley & Sons，1999.
[25] GREEN D M，SWETS J A.Signal detection theory and psychophysics[M].New York：Wiley，1966.
[26] MAIRE M，ARBELáEZ P，FOWLKES C，et al.Using contours to detect and localize junctions in natural images[C]//2008 IEEE Conference on Computer Vision and Pattern Recognition，2008：1-8.
[27] SHI J，MALIK J.Normalized cuts and image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2000，22（8）：888-905.
[28] ARBELAEZ P，MAIRE M，FOWLKES C，et al.From contours to regions：an empirical evaluation[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition，2009：2294-2301.
[29] MARTIN D，FOWLKES C，TAL D，et al.A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings Eighth IEEE International Conference on Computer Vision，2001：416-423.
[30] FANO R M.Transmission of information：a statistical theory of communications[J].American Journal of Physics，1961，29（11）：793-794.
[31] DALAL N，TRIGGS B.Histograms of oriented gradients for human detection[C]//2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition（CVPR’05），2005：886-893.
[32] LOWE D G.Distinctive image features from scale-invariant keypoints[J].International Journal of Computer Vision，2004，60（2）：91-110.
[33] LECUN Y，BOSER B，DENKER J，et al.Handwritten digit recognition with a back-propagation network[C]//Advances in Neural Information Processing Systems，1989.
[34] TORRESANI L，SZUMMER M，FITZGIBBON A.Efficient object category recognition using classemes[C]//European Conference on Computer Vision.Berlin，Heidelberg：Springer，2010：776-789.
[35] ZEILER M D，TAYLOR G W，FERGUS R.Adaptive deconvolutional networks for mid and high level feature learning[C]//2011 International Conference on Computer Vision，2011：2018-2025.
[36] REN X，BO L.Discriminatively trained sparse code gradients for contour detection[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems，2012：584-592.
[37] ZHANG Z，XING F，SHI X，et al.Semicontour：a semi-supervised learning approach for contour detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：251-259.
[38] GANIN Y，LEMPITSKY V.N4-fields：neural network nearest neighbor fields for image transforms[C]//Asian Conference on Computer Vision.Cham：Springer，2014：536-551.
[39] SHEN W，WANG X，WANG Y，et al.DeepContour：a deep convolutional feature learned by positive-sharing loss for contour detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：3982-3991.
[40] LIM J J，ZITNICK C L，DOLLáR P.Sketch tokens：a learned mid-level representation for contour and object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2013：3158-3165.
[41] KRIZHEVSKY A，SUTSKEVER I，HINTON G E.Imagenet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems，2012.
[42] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409. 1556，2014.
[43] BERTASIUS G，SHI J，TORRESANI L.DeepEdge：a multi-scale bifurcated deep network for top-down contour detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：4380-4389.
[44] BERTASIUS G，SHI J，TORRESANI L.High-for-low and low-for-high：efficient boundary detection from deep object features and its applications to high-level vision[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：504-512.
[45] MANINIS K K，PONT-TUSET J，ARBELáEZ P，et al.Convolutional oriented boundaries：from image segmentation to high-level tasks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，40（4）：819-833.
[46] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[47] LAFFERTY J，MCCALLUM A，PEREIRA F C N.Conditional random fields：probabilistic models for segmenting and labeling sequence data[C]//Proceedings of the Eighteenth International Conference on Machine Learning，2001：282-289.
[48] MNIH V，HEESS N，GRAVES A.Recurrent models of visual attention[C]//Advances in Neural Information Processing Systems，2014.
[49] XU D，OUYANG W，ALAMEDA-PINEDA X，et al.Learning deep structured multi-scale features using attention-gated CRFs for contour prediction[C]//Advances in Neural Information Processing Systems，2017.
[50] LIU Y，CHENG M M，HU X，et al.Richer convolutional features for edge detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：3000-3009.
[51] MA W，GONG C，XU S，et al.Multi-scale spatial context-based semantic edge detection[J].Information Fusion，2020，64：238-251.
[52] XUAN W，HUANG S，LIU J，et al.FCL-Net：towards accurate edge detection via fine-scale corrective learning[J].Neural Networks，2022，145：248-259.
[53] AL-AMAREN A，AHMAD M O，SWAMY M N S.RHN：a residual holistic neural network for edge detection[J].IEEE Access，2021，9：74646-74658.
[54] WANG Y，WANG L，QIU J，et al.Multi-residual connection network for edge detection[J].Neural Processing Letters，2021，53（3）：2165-2174.
[55] HU D，YANG H，HOU X.Distance field-based convolutional neural network for edge detection[J].Computational Intelligence and Neuroscience，2022：1712258.
[56] LIU Y，CHENG M M，FAN D P，et al.Semantic edge detection with diverse deep supervision[J].International Journal of Computer Vision，2022，130（1）：179-198.
[57] CAO Y J，LIN C，LI Y J.Learning crisp boundaries using deep refinement network and adaptive weighting loss[J].IEEE Transactions on Multimedia，2020，23：761-771.
[58] 李正，贺赛先.基于新型网络的多层次边缘检测[J/OL].激光杂志：1-7[2022-10-24].http：//kns.cnki.net/kcms/detail/50.1085.tn.20220929.1906.002.html.
LI Z，HE S X.A novel network for multi-level edge detection[J/OL].Laser Journal：1-7[2022-10-24].http：//kns.cnki.net/kcms/detail/50.1085.tn.20220929.1906.002.html.
[59] 王兵，黄刚，张兴鹏.融合卷积特征的清晰边缘检测研究[J/OL].计算机科学与探索：1-15[2022-10-24].http：//kns.cnki.net/kcms/detail/11.5602.TP.20220902.1615.002.html.
WANG B，HUANG G，ZHANG X P.Research on crisp edge detection based on fusion of convolutional features[J/OL].Journal of Frontiers of Computer Science and Technology：1-15[2022-10-24].http：//kns.cnki.net/kcms/detail/11.5602.TP.20220902.1615.002.html.
[60] 黄胜，冉浩杉.基于语义信息的精细化边缘检测方法[J].计算机工程，2022，48（3）：204-210.
HUANG S，RAN H S.Refined edge detection method based on semantic information[J].Computer Engineering，2022，48（3）：204-210.
[61] 宋杰，于裕，骆起峰.基于RCF的跨层融合特征的边缘检测[J].计算机应用，2020，40（7）：2053-2058.
SONG J，YU Y，LUO Q F.Cross-layer fusion feature based on richer convolutional features for edge detection[J].Journal of Computer Applications，2020，40（7）：2053-2058.
[62] SHI W，CABALLERO J，HUSZáR F，et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：1874-1883.
[63] PINHEIRO P O，LIN T Y，COLLOBERT R，et al.Learning to refine object segments[C]//European Conference on Computer Vision.Cham：Springer，2016：75-91.
[64] HE J，ZHANG S，YANG M，et al.Bi-directional cascade network for perceptual edge detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：3828-3837.
[65] DENG R，SHEN C，LIU S，et al.Learning to predict crisp boundaries[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：562-578.
[66] KELM A P，RAO V S，Z?LZER U.Object contour and edge detection with refine contournet[C]//International Conference on Computer Analysis of Images and Patterns.Cham：Springer，2019：246-258.
[67] LIN G，MILAN A，SHEN C，et al.RefineNet：multi-path refinement networks for high-resolution semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：1925-1934.
[68] SORIA X，RIBA E，SAPPA A.Dense extreme inception network：towards a robust CNN model for edge detection[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision，2020：1923-1932.
[69] CHOLLET F.Xception：deep learning with depthwise sepa-rable convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：1251-1258.
[70] LI K，TIAN Y，WANG B，et al.Bi-directional pyramid network for edge detection[J].Electronics，2021，10（3）：329.
[71] GAO L，ZHOU Z，SHEN H T，et al.Bottom-up and top-down：bidirectional additive net for edge detection[C]//Proceedings of the Twenty-Ninth International Conference on Artificial Intelligence，2021：594-600.
[72] BAO S S，HUANG Y R，XU G Y.Bidirectional multiscale refinement network for crisp edge detection[J].IEEE Access，2022，10：26282-26293.
[73] HUANG G，LIU Z，VAN DER MAATEN L，et al.Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：4700-4708.
[74] RONNEBERGER O，FISCHER P，BROX T.U-net：convolutional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham：Springer，2015：234-241.
[75] LE T，DUAN Y.REDN：a recursive encoder-decoder network for edge detection[J].IEEE Access，2020，8：90153-90164.
[76] SORIA X，POMBOZA-JUNEZ G，SAPPA A D.LDC：lightweight dense CNN for edge detection[J].IEEE Access，2022，10：68281-68290.
[77] AL-AMAREN A，AHMAD M O，SWAMY M N S.A low-complexity residual deep neural network for image edge detection[J].Applied Intelligence，2022：1-18.
[78] DENG X，YANG Y，ZHANG H，et al.PCNN double step firing mode for image edge detection[J].Multimedia Tools and Applications，2022：1-27.
[79] LIN C，ZHANG Z，HU Y.Bio-inspired feature enhancement network for edge detection[J].Applied Intelligence，2022：1-16.
[80] YANG H，LI Y，YAN X，et al.ContourGAN：image contour detection with generative adversarial network[J].Knowledge-Based Systems，2019，164：21-28.
[81] RADFORD A，METZ L，CHINTALA S.Unsupervised representation learning with deep convolutional generative adversarial networks[J].arXiv：1511.06434，2015.
[82] SINDEL A，MAIER A，CHRISTLEIN V.Art2Contour：salient contour detection in artworks using generative adversarial networks[C]//2020 IEEE International Conference on Image Processing（ICIP），2020：788-792.
[83] PU M，HUANG Y，LIU Y，et al.EDTER：edge detection with transformer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：1402-1412.
[84] MOTTAGHI R，CHEN X，LIU X，et al.The role of context for object detection and semantic segmentation in the wild[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2014：891-898.
[85] MéLY D A，KIM J，MCGILL M，et al.A systematic comparison between visual cues for boundary detection[J].Vision Research，2016，120：93-107.
[86] WANG P，YUILLE A.Doc：deep occlusion estimation from a single image[C]//European Conference on Computer Vision.Cham：Springer，2016：545-561.
[87] DOSOVITSKIY A，BEYER L，KOLESNIKOV A，et al.An image is worth 16×16 words：transformers for image recognition at scale[J].arXiv：2010.11929，2020.
[88] SANCHEZ-LENGELING B，REIF E，PEARCE A，et al.A gentle introduction to graph neural networks[J].Distill，2021，6（9）：e33.