Comprehensive Review of Multimodal Data Fusion Methods for Construction Robot Localization

doi:10.3778/j.issn.1002-8331.2402-0008

Abstract

Abstract: The types of localization data sources for construction robots are diverse, and the fusion of multimodal data not only helps to improve the localization performance of construction robots in building projects but also facilitates their collaborative operations. Data fusion methods aim to enhance construction robot localization and data sharing by leveraging the advantages of different data sources, improving data collection and processing methods, this supports the improvement of construction robot localization accuracy, real-time performance, and robustness, ultimately enhancing overall construction efficiency and project management. While there are existing research outcomes on specific scenarios exploring data fusion methods for construction robot positioning, there is currently no comprehensive review article on this topic. Through systematic retrieval, this paper first categorizes the analysis into two types based on whether it integrates with prior data：fusion of prior data and real-time sensor data fusion, and fusion of data from multiple sensors. Subsequently, a comparative analysis of data fusion methods is conducted. Finally, the paper summarizes and anticipates the future research directions of multimodal data fusion methods for construction robots. The analysis of current research results reveals significant variations in the choice and effectiveness of positioning data sources for construction robots. This review can serve as a reference for further research in related fields.

Key words: construction robot, localization, data fusion method, multimodal data

摘要： 施工机器人的定位数据源种类繁多，融合多模态数据不仅有助于提升建筑项目中施工机器人的定位性能，同时也方便施工机器人的协同作业。数据融合方法旨在通过不同数据源的优势互补，改进数据采集及处理方法等，实现施工机器人的定位和数据共享，支持施工机器人定位精度、实时性或鲁棒性等的提高，从而提高整体建筑施工效率和项目管理水平。已有不少针对特定场景探索施工机器人定位的数据融合方法相关研究成果，但尚无针对施工机器人定位的数据融合方法相关研究综述。经系统的检索，首先，按照是否与先验数据融合，将其分为先验数据与传感器实时数据融合和多种传感器数据融合两类进行分析；然后，对数据融合方法进行对比分析；最后，总结和展望了施工机器人多模态数据融合方法的未来研究方向。从研究结果分析，现阶段已有的研究成果中，施工机器人定位的数据源选择差异性较大，定位效果差异也很大。该综述可为相关领域的进一步研究提供参考。

关键词: 施工机器人, 定位, 数据融合方法, 多模态数据

LI Jiayi, MA Zhiliang, CHENG Lijie, JI Xinlin. Comprehensive Review of Multimodal Data Fusion Methods for Construction Robot Localization[J]. Computer Engineering and Applications, 2024, 60(15): 11-23.

李佳益, 马智亮, 陈礼杰, 季鑫霖. 面向施工机器人定位的多模态数据融合方法研究综述[J]. 计算机工程与应用, 2024, 60(15): 11-23.

References

[1] TONG S H, XU W, ZHANG X C, et al. Experimental and theoretical analysis on truss construction robot: automatic grasping and hoisting of concrete composite floor slab[J]. Journal of Field Robotics, 2023, 40: 272-288.
[2] DING L Y, JIANG W G, ZHOU Y, et al. BIM-based task-level planning for robotic brick assembly through image-based 3D modeling[J]. Advanced Engineering Informatics, 2020, 43: 100993.
[3] JI J J, SHAN J, MISYURIN S Y, et al. Localization on a-prior information of plane extraction[J]. Plos One, 2023, 18(5).
[4] CHEN J J, LI S, LU W S. Align to locate registering photogrammetric point clouds to BIM for robust[J]. Building and Environment, 2022, 209: 1-14.
[5] 帅子沛. 室内建筑抹灰机器人智能导航研究[D]. 成都: 电子科技大学, 2022.
SHUAI Z P. Research on intelligent navigation of plastering robots for indoor buildings[D]. Chengdu: University of Electronic Science and Technology of China, 2022.
[6] ZHANG Z J, CHENG X J, YANG B L, et al. Exploration of indoor barrier-free plane intelligent lofting system combining BIM and multi-sensors[J]. Remote Sensing, 2020, 12(20): 3306.
[7] LUNDEEN K M, KAMAT V R, MENASSA C, et al. Autonomous motion planning and task execution in geometrically adaptive robotized construction work[J]. Automation in Construction, 2019, 100: 24-45.
[8] 程时伟. 人机交互概论[M]. 杭州: 浙江大学出版社, 2018.
CHENG S W. Introduction to human-computer interaction[M]. Hanghzou: Zhejiang University Press, 2018.
[11] HUANG J H, JUNGINGER S, LIU H, et al. Indoor positioning systems of mobile robots a review[J]. Robots, 2023, 12(2): 47.
[12] 李学友. IMU/DGPS 辅助航空摄影测量原理、方法及实践[D]. 郑州: 中国人民解放军信息工程大学, 2005.
LI X Y. IMU/DGPS-supported photogrammetry-theory, approaches and practice, institute of surveying and mapping[D]. Zhengzhou: Information Engineering University, 2005.
[13] SHEN D, XU Y H, HUANG Y K. Research on 2D-SLAM of indoor mobile robot based on laser radar[C]//Proceedings of the 2019 4th International Conference on Automation, Control and Robotics Engineering, Shenzhen, China, July 19-21, 2019.
[14] FU Q, YU H S, LAI L H, et al. A robust RGB-D SLAM system with points and lines for low texture indoor environments[J]. IEEE Sensors Journal, 2019, 19(21): 9908-9920.
[15] YANG N, STUMBERG L V, WANG R, et al. D3VO: deep depth, deep pose and deep uncertainty for monocular visual odometry[C]//Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), 2020: 1278-1289.
[16] SURIYA D. M, SRIVENKATA K S, SUNDARRAJAN G K S, et al. A robust approach for improving the accuracy of IMU based indoor mobile robot localization[C]//Proceedings of the 13th International Conference on Informatics in Control, 2016: 436-445.
[17] ILYAS M, KHAW H Y, SELVARAJ N M, et al. Robot-assisted object detection for construction automation data and information-driven approach[J]. IEEE/ASME Transaction on Mechatronics, 2021, 26(6): 2845-2856.
[18] 吴海兵. 基于RFID相位特征的机器人定位方法研究[D]. 武汉: 华中科技大学, 2020.
WU H B. Research of phase-based RFID localization method for robot[D]. Wuhan: Huazhong University of Science and Technology, 2020.
[19] LIU J, LI T R, XIE P, et al. Urban big data fusion based on deep learning: an overview[J]. Information Fusion, 2020, 53: 123-133.
[20] 张毅, 杜凡宇, 罗元, 等. 一种融合激光和深度视觉传感器的SLAM地图创建方法[J]. 计算机应用研究, 2016, 33(10): 2970-2972.
ZHANG Y, DU F Y, LUO Y, et al. Map-building approach based on laser and depth visual sensor fusion SLAM[J]. Application Research of Computers, 2016, 33(10): 2970-2972.
[21] 张红, 程传祺, 徐志刚, 等. 基于深度学习的数据融合方法研究综述[J]. 计算机工程与应用, 2020, 56(24): 1-11.
ZHANG H, CHENG C Q, XU Z G, et al. Survey of data fusion based on deep learning[J]. Computer Engineering and Applications, 2020, 56(24): 1-11.
[22] WU J, SU Y H, CHENG Y W et al. Multi-sensor information fusion for remaining useful life prediction of machining tools by adaptive network based fuzzy inference system[J]. Applied Soft Computing, 2018, 68: 13-23.
[23] 吴艳. 多传感器数据融合算法研究[D]. 西安: 西安电子科技大学, 2004.
WU Y. Revisions to the JDL data fusion model[D]. Xi’an: Xidian University, 2004.
[24] XU Z, GUO S, SONG T, ZENG L D. Localization of mobile robot aided for large-scale construction based on optimized artificial landmark map in ongoing scene[J]. Computer Mode-
ling in Engineering & Sciences, 2022(3): 1853-1882.
[25] 杨清梅, 王立权, 王岚, 等. 基于数据融合的拱泥机器人定位方法[J]. 哈尔滨工程大学学报, 2003, 24(4): 363-367.
YANG Q M, WANG L Q, WANG L, et al. Location of move-in-mud robot based on data fusion[J]. Journal of Harbin Engineering University, 2003, 24(4): 363-367.
[26] 艾青林, 刘刚江, 徐巧宁. 动态环境下基于改进几何与运动约束的机器人RGB-D SLAM算法[J]. 机器人, 2021, 43(2): 167-176.
AI Q L, LIU G J, XU Q N, et al. An RGB-D SLAM algorithm for robot based on the improved geometric and motion constraints in dynamic environment[J]. Robot, 2021, 43(2): 167-176.
[27] 汪生浩, 于铭铭, 朱猛猛, 等. 应用于移动机器人的钢筋交叉点视觉检测系统研究[J]. 工业控制计算机, 2023, 36(6): 83-85.
WANG S H, YU M M, ZHU M M, et al. Research on visual inspection system of rebar intersection applied to mobile robot[J]. Industrial Control Computer, 2023, 36(6): 83-85.
[28] 王洲, 杨明欣, 王新媛. 基于多传感器融合的多旋翼无人机近地面定位算法[J].成都信息工程大学学报, 2018(3):261-267.
WANG Z，YANG M X,WANG X Y. Location algorithm based on multi-sensor fusion for multi-rotor aerial vehiclesto flight near the ground[J]. Journal of Chengdu University of Information Technology, 2018(3):261-267.
[29] 李中道, 刘元盛, 常飞翔, 等. 室内环境下UWB与LiDAR融合定位算法研究[J]. 计算机工程与应用, 2021, 57(6): 260-266.
LI Z D, LIU Y S, CHANG F X, et al. Research on UWB and LiDAR fusion positioning algorithm in indoor environment[J]. Computer Engineering and Applications, 2021, 57(6): 260-266.
[30] 李帅鑫, 李广云, 王力, 等. LiDAR/IMU紧耦合的实时定位方法[J], 自动化学报, 2021, 47(6): 1377-1389.
LI S X, LI G Y, WANG L, et al. LiDAR/IMU tightly coupled real-time localization method[J]. Acta Automatica Sinica, 2021, 47(6): 1377-1389.
[31] 董伯麟, 柴旭. 基于IMU/视觉融合的导航定位算法研究[J]. 压电与声光, 2020, 42(5): 724-728.
DONG B L, CHAI X. Research on navigation and localization algorithm based on IMU/vision fusion[J]. Piezoelectricity and Acoustooptics, 2020, 42(5): 724-728.
[32] 章弘凯, 陈年生, 代作晓, 等. 一种多层次数据融合的SLAM定位算法[J]. 机器人, 2021, 43(6): 641-652.
ZHANG H K, CHEN N S, DAI Z X, et al. A multi-level data fusion localization algorithm for SLAM[J]. Robot, 2021, 43(6): 641-652.
[33] 李昀泽. 基于激光雷达的室内机器人SLAM研究[D]. 广州: 华南理工大学, 2017.
LI Y Z. Research on SLAM of indoor robot based on LiDAR[D]. Guangzhou: South China University of Technology, 2017.
[34] 陈小伟. 基于BIM特征的施工机器人定位建图与环境理解方法研究[D]. 天津: 河北工业大学, 2022.
CHENG X W. Research on construction robot localization mapping and environment understanding based on BIM features[D]. Tianjin: Hebei University of Technology, 2022.
[35] PEAVY M, KIM P, OVEDIRAN H, et al. Integration of real-time semantic building map updating with adaptive monte carlo localization (AMCL) for robust indoor mobile robot localization[J]. Advanced Engineering Informatics, 2023, 13(2): 909.
[36] LI C, ZHUANG Y, YAN F. Deep sensor fusion between 2D laser scanner and IMU for mobile robot localization[J]. IEEE Sensors Journal, 2021, 21(6): 8501-8509.
[37] LI H J, MAO Y, YOU W, et al. A neural network approach to indoor mobile robot localization[C]//Proceedings of the 19th International Symposium on Distributed Computing and Applications for Business Engineering and Science, 2020.
[38] ZHAO X, CHEAH C C. BIM-based indoor mobile robot initialization for construction automation using object detection[J]. Automation in Construction, 2022, 146: 104647.
[39] SCHAUB L, PODKOSOVA L, SCHOENAUER C, et al. Point cloud to BIM registration for robot localization and augmented reality[C]//Proceedings of the IEEE International Symposium on Mixed and Augmented Reality, 2022: 77-84.
[40] MOURA M S, RIZZO C, SERRANO D. BIM-based loca-
lization and mapping for mobile robots in construction[C]//Proceedings of the IEEE International Conference on Autonomous Robot Systems and Competitions, 2021: 12-18.
[41] 刘今越, 陈小伟, 贾晓辉, 等. BIM校正累计误差的激光里程计求解方法[J]. 仪器仪表学报, 2022, 43(1): 93-102.
LIU J Y, CHENG X W, JIA X H, et al. The solution method of laser odometer for BIM correction of cumulative error[J]. Chinese Journal of Scientific Instrument, 2022, 43(1): 93-102.
[42] CHEN Y, PERVAN B. Landmark augmentation for mobile robot localization safety[J]. IEEE Robotics and Automation Letters, 2021, 6(1): 119-126.
[43] HALDER S, AFSARI K, KOWSKI J S, et al. Real-time and remote construction progress monitoring with a quadruped robot using augmented reality[J]. Buildings, 2022, 12(11).
[44] 焦传佳. 基于Apriltag图像识别的移动机器人定位研究[J]. 电子测量与仪器学报, 2021, 35(1): 110-119.
JIAO C J. Research on positioning of mobile robot based on AprilTag image recognition[J]. Journal of Electronic Measurement and Instrumentation, 2021, 35(1): 110-119.
[45] KAYHANI N, MCCABE B, ABDELAAL A, et al. Tag-based indoor localization of UAVs in construction environments: opportunities and challenges in practice[C]//Proceedings of the Construction Research Congress, 2020: 226-235.
[46] KAYHANIA N, ZHAO W, MCCABE B, et al. Tag-based visual-inertial localization of unmanned aerial vehicles in indoor construction environments using an on-manifold extended Kalman filter[J]. Automation in Construction, 2022, 135: 1-20.
[47] LI Z Q, HUANG J D. Study on the use of Q-R codes as landmarks for indoor positioning preliminary results[C]//Proceedings of the IEEE/ION Position, Location and Navigation Symposium, 2018: 1270-1276.
[48] 王家恩, 肖献强. 基于 QR 码视觉定位的移动机器人复合导航方法研究[J]. 仪器仪表学报, 2018, 39(8): 230-238.
WANG J N, XIAO X Q. Mobile robot integrated navigation method based on QR code vision positioning[J]. Chinese Journal of Scientific Instrument, 2018, 39(8): 230-238.
[49] 郭金迪. 面向建筑环境的自主打磨移动机器人研究[D]. 沈阳: 东北大学, 2020.
GUO J D. Research on autonomous polishing mobile robot for building environment[D]. Shenyang: Northeastern University, 2020.
[50] 张帅. 地砖铺设机器人系统设计及运动规划研究[D]. 南京: 南京航空航天大学, 2020.
ZHANG S. Research on design and motion planning of robotic floor-tiling system[D]. Nanjing: Nanjing University of Aeronautics and Astronautics, 2020.
[51] LI X, JIANG X, LIU Y. Construction robot localization system based on multi-sensor fusion and 3D construction drawings[C]//Proceedings of the International Conference on Robotics and Biomimetics, 2021: 27-31.
[52] KIM P, PARK J, CHO Y K, et al. UAV-assisted autonomous mobile robot navigation for as-is 3D data collection and registration in cluttered environments[J]. Automation in Construction, 2019, 106: 102918.
[53] KIM P, PARK J, CHO Y K. As-is geometric data collection and 3D visualization through the collaboration between UAV and UGV[C]//Proceedings of the International Symposium on Automation and Robotics in Construction, 2019.
[54] 王舜. 瓷砖铺贴机器人的瓷砖三维空间定位方法研究 [J]. 国外电子测量技术, 2022, 41(10): 85-91.
WANG S. Design of tile three-dimensional space positioning system for floor-tiling robot[J]. Foreign Electronic Measurement Technology, 2022, 41(10): 85-91.
[55] 许镟. 无人推土机系统位姿估计与室外场景建图技术研究[D]. 武汉: 华中科技大学, 2022.
XU X. Research on pose estimation and outdoor scene mapping technology of unmanned bulldozer system[D]. Wuhan: Huazhong University of Science and Technology, 2022.
[56] 赵洋, 刘国良, 田国会, 等. 基于深度学习的视觉SLAM综述[J]. 机器人, 2017, 39(6): 889-896.
ZHAO Y, LIU G L, TIAN G H, et al. A survey of visual SLAM based on deep learning[J]. Robot, 2017, 39(6): 889-896.
[57] MENG T, JING X Y, YAN Z, et al. A survey on machine learning for data fusion[J]. Information Fusion, 2020, 57: 115-129.
[58] 程志伟. 基于力反馈和视觉的板材安装定位研究[D]. 天津: 河北工业大学, 2010.
CHENG Z W. Research on slabstone installation and positioning based on force feedback and vision[D]. Tianjin: Hebei University of Technology, 2010.
[59] 孙建刚, 高娜, 杨冬, 等. 建筑板材安装机器人视觉系统及标定方法研究[J]. 河北工业大学学报, 2013, 42(4): 64-68.
SUN J G, GAO N, YANG D, et al. Research of the building slabstone installing robot vision system and the calibration method[J]. Journal of Hebei University of Technology, 2013, 42(4): 64-68.
[60] WANG W D, ZHAO H P, CHI S Y, et al. A control method for hydraulic manipulators in automatic emulsion filling[J]. Automation in Construction, 2018, 91: 92-99.
[61] ZHAO Q J, LI X F, LU J X, et al. Monocular vision-based parameter estimation for mobile robotic painting[J]. IEEE Transactions on Instrumentation and Measurement, 2018, 68(10): 3589-3599.
[62] TSURUTA T, MIURA K, MIYAGUCHI M. Mobile robot for marking free access floors at construction sites[J]. Automation in Construction, 2019, 107: 102912.

[1]	LI Jun, CAO Lin, ZHANG Fan, DU Kangning, GUO Yanan. Siamese Networks for Object Tracking on Statistical Characteristics of Distributions [J]. Computer Engineering and Applications, 2024, 60(8): 213-224.
[2]	WANG Jiayi, CHEN Ziyang, YUAN Xiaochen, ZHAO Genping. Perceptual Feature Fusion Network for Cross-View Geo-Localization [J]. Computer Engineering and Applications, 2024, 60(3): 255-262.
[3]	YU Junqi, CHEN Yisheng, FENG Chunyong, SU Yucong, GUO Jugang. Review of Research on Local Path Planning for Intelligent Construction Robots [J]. Computer Engineering and Applications, 2024, 60(10): 16-29.
[4]	LIU Mingzhe, XU Guanghui, TANG Tang, QIAN Xiaojian, GENG Ming. Review of SLAM Based on Lidar [J]. Computer Engineering and Applications, 2024, 60(1): 1-14.
[5]	HUANG Zhibin, CHEN Xun. UAV Cluster Forest Fire Detection Method Based on PSO-GA Algorithm [J]. Computer Engineering and Applications, 2023, 59(9): 289-294.
[6]	LIU Jiamin, CHEN Shenglun, WANG Zhihui, LI Haojie. Marker-Assisted Multi-Feature Fusion Localization Method [J]. Computer Engineering and Applications, 2023, 59(6): 187-195.
[7]	PENG Zexin, ZENG Bi, LIU Jianqi. Global Localization Method Based on Global Feature Point Matching [J]. Computer Engineering and Applications, 2023, 59(3): 264-275.
[8]	ZHANG Wenwei, FU Qifan, WANG Yingguan, FU Hengcheng, WEI Zhi, DING Huaze. Improved PSPNet Artillery Flame Segmentation Algorithm and Applications [J]. Computer Engineering and Applications, 2023, 59(23): 154-164.
[9]	GAO Hui, DENG Miaolei, ZHAO Wenjun, CHEN Faquan, ZHANG Dexian. Application of Improved Transformer Based on Weakly Supervised in Crowd Localization [J]. Computer Engineering and Applications, 2023, 59(19): 92-98.
[10]	XU Chundong, LIU Hao, MIN Yuan, ZHEN Yadi. Sound Event Localization and Detection Based on Dual Attention [J]. Computer Engineering and Applications, 2023, 59(19): 99-105.
[11]	GUO Wenbin, YANG Xingming, JIANG Zheyuan, WU Kewei, XIE Zhao. Multi-Temporal Scales Consensus for Weakly Supervised Temporal Action Localization [J]. Computer Engineering and Applications, 2023, 59(10): 151-161.
[12]	CHEN Dan, WU Xin. Research on Improved Laser SLAM Algorithm for Mobile Robots [J]. Computer Engineering and Applications, 2022, 58(4): 163-168.
[13]	WANG Heng, WU Bo, WANG Zhenming, YU Jianfeng. SLAM System Based on Depth Estimation Network SS-Net [J]. Computer Engineering and Applications, 2022, 58(24): 265-275.
[14]	YUAN Saimei, HUANG Yimeng, FENG Lihang, ZHU Wenjun, YI Yang. X_YOWO：Real-Time Human Behavior Positioning Method [J]. Computer Engineering and Applications, 2022, 58(20): 148-156.
[15]	MA Jingxuan, WANG Hongyu, CAO Yan, QIAO Wenchao, HAN Jiaozhi, WU Changxue. Implementation of Densified Mapping of Monocular ORB-SLAM for Embedded Platform [J]. Computer Engineering and Applications, 2022, 58(16): 213-218.