Multi-Sensor Data Fusion for Robotic 3D Target Detection in CPU Environment

doi:10.3778/j.issn.1002-8331.2308-0119

Abstract

Abstract: Real-time and accurate 3D object detection algorithms can provide target position and shape information, ensuring efficient navigation and effective obstacle avoidance for mobile robots, among other tasks. Existing 3D object detection algorithms heavily rely on the computational capabilities of hardware devices. A multi-sensor fusion 3D object detection method that can be deployed in mobile robot CPU environments has been proposed to reduce hardware requirements while ensuring detection accuracy. The method combines 2D object detection and point cloud clustering techniques. It utilizes 2D object detection technology to obtain object detection information from images, then performs point cloud segmentation within the detection bounding boxes based on the spatial mapping relationship between cameras and lidars. The segmented point clouds are further clustered and processed for information extraction, achieving 3D object detection and localization capabilities. By comparing with the classical multi-sensor 3D target detection algorithm MVX-Net, the algorithm in this paper has better detection accuracy with smaller computational complexity. Furthermore, the method is deployed and analyzed on actual mobile robot CPU devices at the edge terminal, achieving a processing speed of 0.069 seconds per frame, satisfying the 10 Hz laser radar frequency requirement.

Key words: 3D object detection, multi-sensor data fusion, CPU, mobile robot

摘要： 实时、准确的3D目标检测算法能提供目标的位置和形态信息，为移动机器人实现高效导航、有效避障等各项任务提供保障。现有的3D目标检测算法对硬件设备运算能力的依赖较为严重，为了在确保检测精度的同时降低方法对硬件设备的要求，提出一种能部署在移动机器人CPU环境下的多传感器融合3D目标检测方法。方法结合了2D目标检测和点云聚类技术，利用2D目标检测技术从图像中获取目标的检测信息，根据相机与雷达的空间映射关系对检测框内的点云进行分割，并对分割后的点云进行聚类和信息提取，从而实现3D目标的检测和定位功能。通过与经典的多传感器3D目标检测算法MVX-Net的对比，该算法有更优的检测精度，同时具有更小的计算复杂度。此外，该方法在实际移动机器人CPU设备的边缘终端上进行部署分析，算法的处理速度达到0.069 s/帧，满足10 Hz激光雷达频率的需求。

关键词: 3D目标检测, 多传感器数据融合, CPU, 移动机器人

LOU Jin, LIU Enbo, TANG Wei, ZHANG Renyuan. Multi-Sensor Data Fusion for Robotic 3D Target Detection in CPU Environment[J]. Computer Engineering and Applications, 2024, 60(19): 120-129.

楼进, 刘恩博, 唐炜, 张仁远. CPU环境下多传感器数据融合的机器人3D目标检测方法[J]. 计算机工程与应用, 2024, 60(19): 120-129.

References

[1] GIRSHICK R. Fast R-CNN[C]//Proceedings of the IEEE International Conference on Computer Vision, 2015: 1440-1448.
[2] HE K, GKIOXARI G, DOLLáR P, et al. Mask R-CNN[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 2961-2969.
[3] 王琳毅, 白静, 李文静, 等. YOLO系列目标检测算法研究进展[J]. 计算机工程与应用, 2023, 59(14): 15-29.
WANG L Y, BAI J, LI W J, et al. Research progress of YOLO series target detection algorithms[J]. Computer Engineering and Applications, 2023, 59(14): 15-29.
[4] GE Z, LIU S, WANG F, et al. YOLOX: exceeding YOLO series in 2021[J]. arXiv:2107. 08430, 2021.
[5] CHEN X, KUNDU K, ZHU Y, et al. 3D object proposals for accurate object class detection[C]//Advances in Neural Information Processing Systems, 2015, 28: 424-432.
[6] CHEN X, KUNDU K, ZHU Y, et al. 3D object proposals using stereo imagery for accurate object class detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(5): 1259-1272.
[7] LI P, CHEN X, SHEN S. Stereo R-CNN based 3D object detection for autonomous driving[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019: 7644-7652.
[8] WANG Y, CHAO W L, GARG D, et al. Pseudo-lidar from visual depth estimation: bridging the gap in 3D object detection for autonomous driving[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019: 8445-8453.
[9] WENG X, KITANI K. Monocular 3D object detection with pseudo-lidar point cloud[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops, 2019: 857-866.
[10] QI C R, SU H, MO K, et al. PointNet: deep learning on point sets for 3D classification and segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 652-660.
[11] LUO W, YANG B, URTASUN R. Fast and furious: real time end-to-end 3D detection, tracking and motion forecasting with a single convolutional net[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 3569-3577.
[12] ZHOU Y, TUZEL O. VoxelNet: end-to-end learning for point cloud based 3D object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 4490-4499.
[13] DENG J, SHI S, LI P, et al. Voxel R-CNN: towards high performance voxel-based 3D object detection[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(2): 1201-1209.
[14] QI C R, YI L, SU H, et al. PointNet++: deep hierarchical feature learning on point sets in a metric space[C]//Advances in Neural Information Processing Systems, 2017, 30: 5100-5109.
[15] YANG Z, SUN Y, LIU S, et al. 3DSSD: point-based 3D single stage object detector[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020: 11040-11048.
[16] ZHOU S, TIAN Z, CHU X, et al. FastPillars: a deployment-friendly pillar-based 3D detector[J]. arXiv:2302.02367, 2023.
[17] CHEN X, MA H, WAN J, et al. Multi-view 3D object detection network for autonomous driving[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 1907-1915.
[18] KU J, MOZIFIAN M, LEE J, et al. Joint 3D proposal generation and object detection from view aggregation[C]//Proceedings of the IEEE International Conference on Intelligent Robots and Systems, 2018: 5750-5757.
[19] SINDAGI V A, ZHOU Y, TUZEL O. MVX-Net: multimodal VoxelNet for 3D object detection[C]//Proceedings of the International Conference on Robotics and Automation, 2019: 7276-7282.
[20] YIN T, ZHOU X, KRAHENBUHL P. Center-based 3D object detection and tracking[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021: 11784-11793.
[21] GEIGER A, LENZ P, URTASUN R. Are we ready for autonomous driving? the kitti vision benchmark suite[C]// Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2012: 3354-3361.
[22] 刘永刚, 于丰宁, 章新杰, 等. 基于激光点云与图像融合的3D目标检测研究[J]. 机械工程学报, 2022, 58(24): 289-299.
LIU Y G, YU F N, ZHANG X J, et al. Research on 3D object detection based on laser point cloud and image fusion[J]. Journal of Mechanical Engineering, 2022, 58(24): 289-299.
[23] SCHUBERT E, SANDER J, ESTER M, et al. DBSCAN revisited, revisited: why and how you should (still) use DBSCAN[J]. ACM Transactions on Database Systems, 2017, 42(3): 1-21.
[24] 黄远宪, 李必军, 黄琦, 等. 融合相机与激光雷达的目标检测、跟踪与预测[J]. 武汉大学学报(信息科学版), 2024, 49(6): 945-951.?
HUANG Y X, LI B J, HUANG Q, et al. Camera-lidar fusion for object detection, tracking and prediction[J]. Geomatics and Information Science of Wuhan University, 2024,49(6): 945-951.