基于边缘差分信息融合的多模态实时三维目标检测

doi:10.3778/j.issn.1002-8331.2407-0534

摘要/Abstract

摘要： 基于多模态的三维目标检测可以利用点云的几何信息和图像的语义信息。针对多模态三维目标检测中存在的边缘信息无法充分利用、异构数据融合困难、推理速度慢等问题，提出一种高效的基于边缘差分信息融合的多模态实时三维目标检测算法（multimodal real-time 3D object detection based on edge differential information fusion, EDMR-Net）。在融合阶段提出了一个差分特征增强融合模块，通过扩散函数利用图像的差分信息增强点云语义表达以实现异构数据的互补，使用丰富的边缘信息和特征的稳定状态精准定位小目标；利用自适应上下文感知网络对多模态特征进行自适应权重分配，进一步细化多尺度上下文信息；为了提升模型对细节信息的捕获能力，在浅层特征中引入了多尺度交叉轴向注意力机制。在KITTI数据集上进行了大量的实验结果表明，所提方法在速度和准确性上都优于主流方法，有效解决了边缘信息利用不充分和多模态推理速度慢的问题，EDMR-Net在保证简单和中等难度检测性能的前提下大大提高了困难场景的检测性能。

关键词: 三维目标检测, 多模态融合, 边缘差分信息, 扩散函数, 注意力机制

Abstract: Multimodal 3D object detection makes full use of the geometric information of the point cloud and the semantic information of the image. Aiming at the problems in multimodal 3D object detection, such as the inability to make full use of edge information，the difficulty of heterogeneous data fusion and slow inference speed，an efficient multimodal 3D object detection algorithm (multimodal real-time 3D object detection based on edge differential information fusion, EDMR-Net) based on edge differential information is proposed. In the fusion stage, a differential feature enhancement fusion (DEF) module is proposed, which enhances the point cloud semantic expression by using the differential information of the image through the diffusion function to achieve the complementarity of heterogeneous data, and precisely locates the small objects using the rich edge information and the stead condition of the features; and the multimodal features are further refined with the multi-scale context information using the adaptive context awareness (ACA) network with the adaptive weight assignment. In order to enhance the model’s ability to capture detailed information, a multi-scale cross-axis attention mechanism is introduced into the shallow layer features. Experimental results on KITTI dataset show that the proposed method outperforms the mainstream methods in terms of speed and accuracy, effectively solves the problems of inadequate utilization of edge information and slow multimodal inference, and EDMR-Net greatly improves the detection performance for difficult scenes while guaranteeing the detection performance for easy and moderate levels.

Key words: 3D object detection, multimodal fusion, edge differential information, diffusion function, attention mechanism

张芷湉, 赵红东, 张珂, 陈丹, 李俨奇. 基于边缘差分信息融合的多模态实时三维目标检测[J]. 计算机工程与应用, 2025, 61(21): 287-296.

ZHANG Zhitian, ZHAO Hongdong, ZHANG Ke, CHEN Dan, LI Yanqi. Multimodal Real-Time 3D Object Detection Based on Edge Differential Information Fusion[J]. Computer Engineering and Applications, 2025, 61(21): 287-296.

参考文献

[1] DROBNITZKY M, FRIEDERICH J, EGGER B, et al. Survey and systematization of 3D object detection models and methods[J]. The Visual Computer, 2024, 40(3): 1867-1913.
[2] LU B, SUN Y, YANG Z Y, et al. HRNet: 3D object detection network for point cloud with hierarchical refinement[J]. Pattern Recognition, 2024, 149: 110254.
[3] 田枫, 宗内丽, 刘芳, 等. 多模态融合的三维目标检测方法研究[J]. 计算机工程与应用, 2024, 60(13): 113-123.
TIAN F, ZONG N L, LIU F, et al. Research on 3D object detection method based on multi-modal fusion[J]. Computer Engineering and Applications, 2024, 60(13): 113-123.
[4] 李红岩, 徐保庆, 张子扬, 等. 基于全局上下文信息的遥感图像小目标检测[J]. 光学学报, 2024, 44(24): 2428004.
LI H Y, XU B Q, ZHANG Z Y, et al. Small target detection in remote sensing images based on global context information[J]. Acta Optica Sinica, 2024, 44(24): 2428004.
[5] 李文举, 储王慧, 崔柳, 等. 结合图采样和图注意力的3D目标检测方法[J]. 计算机工程与应用, 2023, 59(9): 237-244.
LI W J, CHU W H, CUI L, et al. 3D object detection method combining on graph sampling and graph attention[J]. Computer Engineering and Applications, 2023, 59(9): 237-244.
[6] 李文礼, 喻飞, 石晓辉, 等. BEV特征下激光雷达和单目相机融合的目标检测算法研究[J]. 计算机工程与应用, 2024, 60(11): 182-193.
LI W L, YU F, SHI X H, et al. Research on target detection algorithm for fusion of lidar and monocular camera under BEV features[J]. Computer Engineering and Applications, 2024, 60(11): 182-193.
[7] CHEN X Z, MA H M, WAN J, et al. Multi-view 3D object detection network for autonomous driving[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 6526-6534.
[8] LIANG M, YANG B, WANG S L, et al. Deep continuous fusion for multi-sensor 3D object detection[C]//Proceedings of the European Conference on Computer Vision. Cham: Springer, 2018: 663-678.
[9] LI Z H, CUI Y B, LIN Y, et al. MMF-track: multi-modal multi-level fusion for 3D single object tracking[J]. IEEE Transactions on Intelligent Vehicles, 2024, 9(1): 1817-1829.
[10] CHEN Y K, LI Y W, ZHANG X Y, et al. Focal sparse convolutional networks for 3D object detection[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 5418-5427.
[11] XU D F, ANGUELOV D, JAIN A. PointFusion: deep sensor fusion for 3D bounding box estimation[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 244-253.
[12] SINDAGI V A, ZHOU Y, TUZEL O. MVX-Net: multimodal VoxelNet for 3D object detection[C]//Proceedings of the 2019 International Conference on Robotics and Automation. Piscataway: IEEE, 2019: 7276-7282.
[13] VORA S, LANG A H, HELOU B, et al. PointPainting: sequential fusion for 3D object detection[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 4603-4611.
[14] MA J M, SHI G, LI Y X, et al. MAFF-Net: multi-attention guided feature fusion network for change detection in remote sensing images[J]. Sensors, 2022, 22(3): 888.
[15] HUANG T T, LIU Z, CHEN X W, et al. EPNet: enhancing point features with image semantics for 3D object detection[C]//Proceedings of the 16th European Conference on Computer Vision. Cham: Springer, 2020: 35-52.
[16] YOO J H, KIM Y, KIM J, et al. 3D-CVF: generating joint camera and LiDAR features using cross-view spatial feature fusion for 3D object detection[C]//Proceedings of the 16th European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 720-736.
[17] LANG A H, VORA S, CAESAR H, et al. PointPillars: fast encoders for object detection from point clouds[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 12689-12697.
[18] CHARLES R Q, HAO S, MO K C, et al. PointNet: deep learning on point sets for 3D classification and segmentation[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 77-85.
[19] YOU X, DING M, ZHANG M H, et al. PnPNet: pull-and-push networks for volumetric segmentation with boundary confusion[J]. arXiv:2312.08323, 2023.
[20] SHAKER A, MAAZ M, RASHEED H, et al. SwiftFormer: efficient additive attention for transformer-based real-time mobile vision applications[C]//Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2023: 17379-17390.
[21] SHAO H, ZENG Q S, HOU Q B, et al. MCANet: medical image segmentation with multi-scale cross-axis attention[J]. arXiv:2312.08866, 2023.
[22] GUO M H, LU C Z, HOU Q B, et al. SegNeXt: rethinking convolutional attention design for semantic segmentation[J]. arXiv:2209.08575, 2022.
[23] YAN Y, MAO Y X, LI B. SECOND: sparsely embedded convolutional detection[J]. Sensors, 2018, 18(10): 3337.
[24] GEIGER A, LENZ P, URTASUN R. Are we ready for autonomous driving? The KITTI vision benchmark suite[C]//Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2012: 3354-3361.
[25] ZHOU Y, TUZEL O. VoxelNet: end-to-end learning for point cloud based 3D object detection[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 4490-4499.
[26] LIU Z, ZHAO X, HUANG T T, et al. TANet: robust 3D object detection from point clouds with triple attention[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2020: 11677-11684.
[27] KU J, MOZIFIAN M, LEE J, et al. Joint 3D proposal generation and object detection from view aggregation[C]//Proceedings of the 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway: IEEE, 2018: 1-8.
[28] QI C R, LIU W, WU C X, et al. Frustum PointNets for 3D object detection from RGB-D data[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 918-927.
[29] XIE L, XIANG C, YU Z X, et al. PI-RCNN: an efficient multi-sensor 3D object detector with point-based attentive cont-conv fusion module[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2020: 12460-12467.
[30] ZHANG J, XU D, LI Y S, et al. FusionPillars: a 3D object detection network with cross-fusion and self-fusion[J]. Remote Sensing, 2023, 15(10): 2692.
[31] XIE G T, CHEN Z Y, GAO M, et al. PPF-Det: point-pixel fusion for multi-modal 3D object detection[J]. IEEE Transactions on Intelligent Transportation Systems, 2024, 25(6): 5598-5611.
[32] SONG Z Y, ZHANG G X, XIE J, et al. VoxelNextFusion: a simple, unified and effective voxel fusion framework for multi-modal 3D object detection[J]. arXiv:2401.02702, 2024.
[33] LIU M Y, CHEN Y P, XIE J M, et al. MENet: multi-modal mapping enhancement network for 3D object detection in autonomous driving[J]. IEEE Transactions on Intelligent Transportation Systems, 2024, 25(8): 9397-9410.
[34] SHAO Y X, TAN A H, SUN Z T, et al. PV-SSD: a multi-modal point cloud 3D object detector based on projection features and voxel features[J]. IEEE Transactions on Emerging Topics in Computational Intelligence, 2024, 8(5): 3436-3449.
[35] NIE C, JU Z Y, SUN Z F, et al. 3D object detection and tracking based on lidar-camera fusion and IMM-UKF algorithm towards highway driving[J]. IEEE Transactions on Emerging Topics in Computational Intelligence, 2023, 7(4): 1242-1252.
[36] ZHOU J, LIN T X, GONG Z X, et al. SIANet: 3D object detection with structural information augment network[J]. IET Computer Vision, 2024, 18(5): 682-695.
[37] SHI W J, RAJKUMAR R. Point-GNN: graph neural network for 3D object detection in a point cloud[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 1708-1716.
[38] WANG Z X, JIA K. Frustum ConvNet: sliding Frustums to aggregate local point-wise features for amodal 3D object detection[C]//Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway: IEEE, 2019: 1742-1749.
[39] REDMON J, FARHADI A. YOLOv3: an incremental improvement[J]. arXiv:1804.02767, 2018.
[40] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 770-778.