Research Progress and Prospect of V-SLAM Deep Learning Loop Closure Detection

doi:10.3778/j.issn.1002-8331.2111-0045

Abstract

Abstract: Loop closure detection is an important part of simultaneous localization and mapping（SLAM）, which can reduce the cumulative error caused by mobile robots in position estimation and construction of environmental maps. The traditional method adopts the characteristics of artificial design, but it is easy to be affected by factors such as illumination, weather and viewpoint change in the external environment. With the development of deep learning technology, extensive exploration has been carried out in loop closure detection, and the loop closure detection based on deep learning has strong robustness in complex environment. Through combing the background and development status of loop closure detection, this paper compares and analyzes the basic principles and algorithm characteristics of the current visual SLAM loop closure detection methods from three aspects based on deep convolution neural network, automatic encoder and semantic information, summarizes the applicable scenarios of the three methods from the visual application level, and finally, the challenges and research prospects of loop closure detection in three aspects：natural environment change, multiple moving targets and real-time dynamics are discussed.

Key words: simultaneous localization and mapping, loop closure detection, deep learning, convolutional neural network, automatic encoder, semantic information

摘要： 闭环检测是同步定位与建图（simultaneous localization and mapping，SLAM）中的一个重要组成部分，用于减少移动机器人在位置估计和构建环境地图时产生的累计误差。传统方法采用人工设计的特征，但在外界环境中容易受到光照、天气和视点变化等因素所带来的影响。随着深度学习技术的发展，闭环检测得到广泛的探索，且在复杂环境中基于深度学习的闭环检测具有较强的鲁棒性。通过梳理闭环检测的背景和发展现状，从基于深度卷积神经网络、自动编码器和语义信息三个方面，对目前视觉SLAM（visual-SLAM，V-SLAM）闭环检测方法的基本原理、算法特点进行了对比分析，并从视觉应用层面上总结了三类方法所适用的场景，最后讨论了闭环检测未来在自然环境变化、多移动目标和实时动态三个方面所存在的挑战和研究展望。

关键词: 同步定位与建图, 闭环检测, 深度学习, 卷积神经网络, 自动编码器, 语义信息

GAO Gui, WU Xuanheng, WANG Zhongmei, ZHENG Liang. Research Progress and Prospect of V-SLAM Deep Learning Loop Closure Detection[J]. Computer Engineering and Applications, 2022, 58(11): 47-59.

高贵, 伍宣衡, 王忠美, 郑良. V-SLAM深度学习闭环检测研究进展与展望[J]. 计算机工程与应用, 2022, 58(11): 47-59.

References

[1] DURRANT-WHYTE H，BAILEY T.Simultaneous localization and papping：part I[J].IEEE Robotics & Automation Magazine，2006，13（2）：99-110.
[2] KERL C，STURM J，CREMERS　D.Dense visual slam for RGB-D cameras[C]//2013 IEEE/RSJ International Conference on Intelligent Robots and Systems，2013：2100-2106.
[3] KIM A，EUSTICE　R　M.Perception-driven navigation：active visual slam for robotic area coverage[C]//2013 IEEE International Conference on Robotics and Automation，2013：3196-3203.
[4] XIANG G，TAO Z.Loop closure detection for visual SLAM systems using deep neural networks[C]//2015 34th Chinese Control Conference（CCC），2015：5851-5856.
[5] LOWE D.Distinctive image features from scale-invariant key points[J].International Journal of Computer Vision，2003，20：91-110.
[6] GOOL T.Speeded-up robust features（SURF）[J].Computer Vision and Image Understanding，2008，110：346-359.
[7] ROSTEN E.Machine learning for high-speed corner detection[C]//European Conference on Computer Vision，2006.
[8] RUBLEE　E，RABAUD　V，KONOLIGE K，et al.ORB：an efficient alternative to SIFT or SURF[C]//2011 International Conference on Computer Vision，2011：2564-2571.
[9] CSURKA G，DANCE C R，FAN L，et al.Visual categorization with bags of keypoints[C]//ECCV Workshop on Statistical Learning in Computer Vision，2004：59-74.
[10] PERRONNIN F，DANCE C.Fisher kernels on visual vocabularies for image categorization[C]//IEEE Conference on Computer Vision and Pattern Recognition（CVPR’07），2007：1-8.
[11] DANIILIDIS K，MARAGOS P，PARAGIOS N.Improving the fisher kernel for large-scale image classification[C]//European Conference on Computer Vision.Berlin，Heidelberg：Springer，2010.
[12] GALVEZ-LóPEZ D，TARDOS J D.Bags of binary words for fast place recognition in image sequences[J].IEEE Transactions on Robot，2012，28：1188-1197.
[13] GARCIA-FIDALGO E，ORTIZ A.iBoW-LCD：an appearance-based loop closure detection approach using incremental bags of binary words[J].IEEE Robotics and Automation Letters，2018，3（4）：3051-3057.
[14] OLIVA A，TORRALBA A.Modeling the shape of the scene：a holistic representation of the spatial envelope[J].International Journal of Computer Vision，2001，42（3）：145-175.
[15] CUMMINS M，NEWMAN P.FAB-MAP：probabilistic localization and mapping in the space of appearance[J].The International Journal of Robotics Research，2008，27（6）：647-665.
[16] ZHANG G，LILLY M J，VELA P A.Learning binary features online from motion dynamics for incremental loop-closure detection and place recognition[C]//IEEE International Conference on Robotics & Automation，2016：765-772.
[17] 梁志伟，陈燕燕，朱松豪，等.基于视觉词典的单目视觉闭环检测算法[J].模式识别与人工智能，2013，26（6）：51-60.
LIANG Z W，CHEN Y Y，ZHU S H，et al.Loop closure detection algorithm based on monocular vision using visual dictionary[J].Pattern Recognition and Artificial Intelligence，2013，36（6）：51-60.
[18] THRUN S.Probabilistic robotics[J].Communications of the ACM，2005，45（3）：52-57.
[19] SCHINDLER G，BROWN M，SZELISKI R.City-scale location recognition[C]//IEEE Computer Society Conference on Computer Vision & Pattern Recognition，MN，USA，2007：1-7.
[20] CUMMINS M，NEWMAN P.Probabilistic appearance based navigation and loop closing[C]//Proceedings 2007 IEEE International Conference on Robotics and Automation，Roma，Italy，2007：2042-2048.
[21] CHOW C K.Approximating discrete probability distributions with dependence trees[J].IEEE Transactions on Information Theory，2006，14（3）：462-467.
[22] MADDERN W，MILFORD　M，WYETH　G.CAT-SLAM：probabilistic localization and mapping using a continuous appearance-based trajectory[J].The International Journal of Robotics Research，2012，31：429-451.
[23] BENGIO Y，COURVILLE A，VINCENT P.Representation learning：a review and new perspectives[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2012，35（8）：1798-1828.
[24] KRIZHEVSKY A，SUTSKEVER I，HINTON G E.ImageNet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems，2012：1097-1105.
[25] VINCENT P，LAROCHELLE H，LAJOIE I，et al.Stacked denoising autoencoders：learning useful representations in a deep network with a local denoising criterion[J].Journal of Machine Learning Research，2010，11（12）：3371-3408.
[26] VINEET V，MIKSIK O，LIDEGAARD M，et al.Incremental dense semantic stereo fusion for large-scale semantic scene reconstruction[C]//IEEE International Conference on Robotics and Automation.Piscataway，USA：IEEE，2015：75-82.
[27] HOU Y，ZHANG H，ZHOU S.Convolutional neural network-based image representation for visual loop closure detection[C]//2015 IEEE International Conference on Information and Automation，2015：2238-2245.
[28] JIA Y，SHELHAMER E，DONAHUE J，et al.Caffe：convolutional architecture for fast feature embedding[J].arXiv：1408.5093，2014.
[29] ZHOU B L，LAPEDRIZA A，XIAO J X，et al.Learning deep features for scene recognition using places database[C]//Advances in Neural Information Processing Systems，2014：487-495.
[30] XIA Y，JIE L，LIN Q，et al.Loop closure detection for visual slam using PCANet features[C]//2016 International Joint Conference on Neural Networks（IJCNN），2016：2274-2281.
[31] CHAN T H，JIA K，GAO S，et al.PCANet：a simple deep learning baseline for image classification[J].arXiv：1404.3606，2014.
[32] 何元烈，陈佳腾，曾碧.基于精简卷积神经网络的快速闭环检测方法[J].计算机工程，2018，44（6）：182-187.
HE Y L，CHENG J T，ZHENG B.Fast closed loop detection method based on simplification convolutional neural network[J].Computer Engineering，2018，44（6）：182-187.
[33] ZHOU B L，KHOSLA A，LAPEDRIZA A，et al.Places：an image database for deep scene understanding[J] Joumal of Vision，2017，17（10）：611-622.
[34] 潘锡英，何元烈，孙盛，等.基于图像感兴趣区域的机器人闭环检测算法[J].机器人，2019，41（5）：119-125.
PAN X Y，HE Y L，SUN S，et al.A loop closure detection algorithm for robots based on region proposals of interest of image[J].Robot，2019，41（5）：119-125.
[35] 张学典，顾璋琦，秦晓飞.基于VGG16模型的快速闭环检测算法[J].光学仪器，2019，41（3）：1-7.
ZHANG X D，GU Z Q，QIN X F.VGG16 model-based fast loop closure detection algorithm[J].Optical Instruments，2019，41（3）：1-7.
[36] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409.1556，2014.
[37] CHEN B F，YUAN D，LIU C F，et al.Loop closure detection based on multi-scale deep feature fusion[J].Applied Sciences，2019，9（6）：1120.
[38]　HE K，ZHANG X，REN S，et al.Spatial pyramid pooling in deep convolutional networks for visual recognition[J].IEEE Transactions on Pattern Analysis & Machine Intelligence，2014，37（9）：1904-1916.
[39] 郭纪志，刘凤连，杨馨竹，等.基于深度学习的视觉SLAM闭环检测方法[J].光电子.激光，2021，32（6）：9.
GUO J Z，LIU F L，YANG X Z，et al.The closed loop detection method of vision SLAM based on deep learning[J].Journal of Optoelectronics Laser，2021，32（6）：9.
[40] GAO X，ZHANG T.Unsupervised learning to detect loops using deep neural networks for visual SLAM system[J].Autonomous Robots，2017，41（1）：1-18.
[41] STURM J，ENGELHARD N，ENDRES F，et al.A benchmark for the evaluation of RGB-D SLAM systems[C]//2012 IEEE/RSJ International Conference on Intelligent Robots and Systems（IROS），2012：573-580.
[42] MUJA M，LOWE D G.Fast approximate nearest neighbors with automatic algorithm configuration[C]//International Conference on Computer Vision Theory & Application Vissapp，2009：331-340.
[43] CUMMINS M，NEWMAN P.Appearance-only SLAM at large scale with FAB-MAP 2.0[J].International Journal of Robotics Research，2011，30（9）：1100-1123.
[44] 张云洲，胡航，秦操，等.基于栈式卷积自编码的视觉SLAM闭环检测[J].控制与决策，2019，34（5）：981-988.
ZHANG Y Z，HU H，QIN C，et al.Loop closure detection for visual SLAM based on stacked convolutional autoencoder[J].Control and Decision，2019，34（5）：981-988.
[45] MERRILL N，HUANG G.Lightweight unsupervised deep loop closure[J].arXiv：1805.07703，2018.
[46] WROBEL B P.Multiple view geometry in computer vision[J].Kybernetes，2001，30（9/10）：1333-1341.
[47] DALAL N，TRIGGS B.Histograms of oriented gradients for human detection[C]//IEEE Computer Society Conference on Computer Vision & Pattern Recognition，2005：886-893.
[48] WANG Y，ZELL A.Improving feature-based visual SLAM by semantics[C]//2018 IEEE International Conference on Image Processing，Applications and Systems（IPAS）.Piscataway，NJ，USA：IEEE，2018：7-12.
[49] HU M，LI S，WU J，et al.Loop closure detection for visual slam fusing semantic information[C]//2019 Chinese Control Conference（CCC），2019.
[50] REN S，HE K，GIRSHICK R，et al.Faster R-CNN：towards real-time object detection with region proposal networks[J].IEEE Transactions on Pattern Analysis & Machine Intelligence，2017，39（6）：1137-1149.
[51] 郑冰清，刘启汉，赵凡，等.一种融合语义地图与回环检测的视觉SLAM方法[J].中国惯性技术学报，2020，28（5）：629-637.
ZHENG B Q，LIU Q H，ZHAO F，et al.Loop detection and semantic mapping algorithm fused with semantic information[J].Journal of Chinese Inertial Technology，2020，28（5）：629-637.
[52] REDMON J，FARHADI A.YOLOv3：an incremental improvement[J].arXiv：1804.02767，2018.
[53] PATHAK K，BIRK A，VAKEVIIUS N，et al.Fast registration based on noisy planes with unknown correspondences for 3-D mapping[J].IEEE Transactions on Robotics，2010，26（3）：424-441.
[54] MILFORD M J，WYETH G F.SeqSLAM：visual route-based navigation for sunny summer days and stormy winter nights[C]//2012 IEEE International Conference on Robotics and Automation（ICRA），2012.
[55] SüNDERHAUF N，SHIRAZI S，DAYOUB F，et al.On the performance of ConvNet features for place recognition[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems.Piscataway，USA：IEEE，2015：4297-4304.
[56] NEUBERT P，SüNDERHAUF N，PROTZEL P.Superpixel-based appearance change prediction for long-term navigation across seasons[J].Robotics and Autonomous Systems，2015，69（S1）：15-27.
[57] DENG J，DONG　W，SOCHER　R，et al.ImageNet：a large-scale hierarchical image database[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition（CVPR），Miami，FL，USA，2009：248-255.
[58] LIN T Y，MAIRE M，BELONGIE S，et al.Microsoft COCO：common objects in context[J].arXiv：1405.0312，2014.
[59] Mapillary.Mapillary[EB/OL].[2016-05-15].https：//www.mapillary.com.
[60] SZEGEDY C，LIU W，JIA Y，et al.Going deeper with convolutions[J].arXiv：1409.4842，2014.
[61] XIA Y，LI J，QI L，et al.An evaluation of deep learning in loop closure detection for visual SLAM[C]//2017 IEEE International Conference on Internet of Things（iThings） and IEEE Green Computing and Communications（Green-Com） and IEEE Cyber，Physical and Social Computing（CPSCom） and IEEE Smart Data（SmartData），2017：85-91.
[62] 张括嘉，张云洲，吕光浩，等.基于局部语义拓扑图的视觉SLAM闭环检测[J].机器人，2019，41（5）：649-659.
ZHANG K J，ZHANG Y Z，LV G H，et al.Loop closure detection based on local semantic topology for visual SLAM system[J].Robot，2019，41（5）：649-659.