Global-Local Feature Aggregation for Real Point Cloud Semantic Segmentation

doi:10.3778/j.issn.1002-8331.2401-0348

Abstract

Abstract: Learning effective features from real point cloud scenes is still a challenging problem for point cloud segmentation work. A real point cloud semantic segmentation network based on global-local feature aggregation?is presented to improve the capability to identify local fine-grained structures of point clouds and consider global context information. To learn the local fine-grained structures of point clouds, automatically modify the point cloud grouping scale, and improve the network’s feature learning adaptability to unevenly dense point clouds, a density-adaptive local neighborhood feature extraction layer is firstly?presented, which is based on the idea of hierarchical feature extraction in Pointnet++. Secondly, a spatial attention module is added to the encoder and decoder, which uses the self-attention mechanism to learn the correlation between global points and uses a residual connection to mitigate the vanishing gradient issue. In addition, improved spatial coding is adopted to explicitly express the spatial structure and optimize network parameters. Finally, experiments are conducted on the public data sets S3DIS and Semantic3D, in which the mIoU (mean intersection over union) and OA (overall accuracy) tested in S3DIS are improved by 16.9?percentage points and 7.4?percentage points respectively compared with the baseline PointNet++.

Key words: point cloud processing, semantic segmentation, self-attention, density adaptation, local feature encoding

摘要： 从真实点云场景中学习有效特征进行点云分割仍是具有挑战性的难题。为加强点云局部细粒度特征的提取能力并兼顾全局上下文信息，提出聚合全局-局部特征的真实点云语义分割网络。基于Pointnet++网络分层提取特征的思想，引入密度自适应局部邻域特征提取层学习点云局部细粒度特征，自动调节点云分组尺度，增强网络对于疏密不均点云的特征学习适应性。在编码器和解码器中加入空间注意力模块，通过自注意力机制学习全局点间的相关性并通过残差连接缓解梯度消失问题。此外，采用了改进的空间编码，显式地表达空间结构，优化网络参数。在公开数据集S3DIS和Semantic3D上进行实验，其中在S3DIS测试的平均交并比（mIoU）和总体准确率（OA）分别比基准PointNet++提高了16.9个百分点和7.4个百分点。

关键词: 点云处理, 语义分割, 自注意力, 密度自适应, 局部特征编码

TIAN Yuan, ZHAO Mingfu, SONG Tao, XIONG Hailong, YE Dingxing, WANG Min. Global-Local Feature Aggregation for Real Point Cloud Semantic Segmentation[J]. Computer Engineering and Applications, 2025, 61(8): 260-266.

田媛, 赵明富, 宋涛, 熊海龙, 叶定兴, 王敏. 聚合全局-局部特征的真实点云语义分割[J]. 计算机工程与应用, 2025, 61(8): 260-266.

References

[1] 武越, 苑咏哲, 向本华, 等. 三维点云配准中的计算智能方法综述[J]. 中国图象图形学报, 2023, 28(9): 2763-2787.
WU Y, YUAN Y Z, XIANG B H, et al. Overview of the computational intelligence method in 3D point cloud registration[J]. Journal of Image and Graphics, 2023, 28(9): 2763-2787.
[2] 袁萌萌, 张泽旭. 三维点云与图像双模态融合的空间目标部件识别方法[J]. 宇航学报, 2023, 44(5): 796-804.
YUAN M M, ZHANG Z X. A recognition method for components of space target based on bi-modal fusion of 3D point cloud and image[J]. Journal of Astronautics, 2023, 44(5): 796-804.
[3] 赵佳琦, 周勇, 何欣, 等. 基于深度学习的点云分割研究进展分析[J]. 电子与信息学报, 2022, 44(12): 4426-4440.
ZHAO J Q, ZHOU Y, HE X, et al. Research progress analysis of point cloud segmentation based on deep learning[J]. Journal of Electronics & Information Technology, 2022, 44(12): 4426-4440.
[4] 王大方, 尚海, 曹江, 等. 基于自注意力机制的自动驾驶场景点云语义分割方法[J]. 汽车工程, 2022, 44(11): 1656-1664.
WANG D F, SHANG H, CAO J, et al. Semantic segmentation method of point cloud in automatic driving scene based on self-attention mechanism[J]. Automotive Engineering, 2022, 44(11): 1656-1664.
[5] 庞磊, 曹志强, 喻俊志. 基于A*和TEB融合的行人感知无碰跟随方法[J]. 航空学报, 2021, 42(4): 501-510.
PANG L, CAO Z Q, YU J Z. A pedestrian-aware collision-free following approach for mobile robots based on A* and TEB[J]. Acta Aeronautica et Astronautica Sinica, 2021, 42(4): 501-510.
[6] PEREZ-PEREZ Y, GOLPARVAR-FARD M, EL-RAYES K. Segmentation of point clouds via joint semantic and geometric features for 3D modeling of the built environment[J]. Automation in Construction, 2021, 125(7): 103584.
[7] 卢健, 贾旭瑞, 周健, 等. 基于深度学习的三维点云分割综述[J]. 控制与决策, 2023, 38(3): 595-611.
LU J, JIA X R, ZHOU J, et al. A review of deep learning based on 3D point cloud segmentation[J]. Control and Decision, 2023, 38(3): 595-611.
[8] MINAEE S, BOYKOV Y, PORIKLI F, et al. Image segmentation using deep learning: a survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(7): 3523-3542.
[9] HUANG J, YOU S. Point cloud labeling using 3D convolutional neural network[C]//Proceedings of the 23rd International Conference on Pattern Recognition, 2016: 2670-2675.
[10] MENG H Y, GAO L, LAI Y K, et al. VV-Net: voxel VAE net with group convolutions for point cloud segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 8499-8507.
[11] WU B, WAN A, YUE X, et al. SqueezeSeg: convolutional neural nets with recurrent CRF for real-time road-object segmentation from 3D Lidar point cloud[C]//Proceedings of the IEEE International Conference on Robotics and Automation, 2018: 1887-1893.
[12] BELTRáN J, GUINDEL C, MORENO F M, et al. BirdNet: a 3D object detection framework from LiDAR information[C]//Proceedings of the 21st International Conference on Intelligent Transportation Systems, 2018: 3517-3523.
[13] HUANG Q, WANG W, NEUMANN U. Recurrent slice networks for 3D segmentation of point clouds[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 2626-2635.
[14] DAI A, CHANG A X, SAVVA M, et al. ScanNet: richly-annotated 3D reconstructions of indoor scenes[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 5828-5839.
[15] ARMENI I, SENER O, ZAMIR A R, et al. 3D semantic parsing of large-scale indoor spaces[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1534-1543.
[16] CHANG A, DAI A, FUNKHOUSER T, et al. Matterport3D: learning from RGB-D data in indoor environments[C]//Proceedings of the International Conference on 3D Vision, 2017.
[17] BEHLEY J, GARBADE M, MILIOTO A, et al. SemanticKITTI: a dataset for semantic scene understanding of Lidar sequences[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 9297-9307.
[18] HACKEL T, SAVINOV N, LADICKY L, et al. Semantic3D. net: a new large-scale point cloud classification benchmark[J]. arXiv:1704.03847, 2017.
[19] LI X, LI C, TONG Z, et al. Campus3D: a photogrammetry point cloud benchmark for hierarchical understanding of outdoor scene[C]//Proceedings of the 28th ACM International Conference on Multimedia, 2020: 238-246.
[20] QI C R, SU H, MO K, et al. PointNet: deep learning on point sets for 3D classification and segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 652-660.
[21] QI C R, YI L, SU H, et al. PointNet++: deep hierarchical feature learning on point sets in a metric space[J]. arXiv:1706.24013, 2017 .
[22] WANG Y, SUN Y, LIU Z, et al. Dynamic graph CNN for learning on point clouds[J]. ACM Transactions on Graphics, 2019, 38(5): 1-12.
[23] ZHAO H, JIANG L, FU C W, et al. PointWeb: enhancing local neighborhood features for point cloud processing[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 5565-5573.
[24] LANDRIEU L, SIMONOVSKY M. Large-scale point cloud semantic segmentation with superpoint graphs[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 4558-4567.
[25] WU W X, QI Z, LI F X. PointConv: deep convolutional networks on 3D point clouds[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 9621-9630.
[26] THOMAS H, QI C R, DESCHAUD J E, et al. KPConv: flexible and deformable convolution for point clouds[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 6411-6420.
[27] LI Y, BU R, SUN M, et al. PointCNN: convolution on x-transformed points[J]. arXiv:1801.07791, 2018.
[28] LIU X, HAN Z, LIU Y S, et al. Point2Sequence: learning the shape representation of 3D point clouds with an attention-based sequence to sequence network[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2019: 8778-8785.
[29] CHEN L, CHEN W, XU Z, et al. DAPNet: a double self-attention convolutional network for point cloud semantic labeling[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14: 9680-9691.
[30] HU Q, YANG B, XIE L, et al. RandLA-Net: efficient semantic segmentation of large-scale point clouds[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 11108-11117.
[31] RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, 2015: 234-241.
[32] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 6000-6010.
[33] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
[34] SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethinking the inception architecture for computer vision[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 2818-2826.
[35] LOSHCHILOV I, HUTTER F. Decoupled weight decay regularization[J]. arXiv:1711.05101, 2017.
[36] LI G, MULLER M, THABET A, et al. DeepGCNs: can GCNs go as deep as CNNs?[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 9267-9276.
[37] YANG J, ZHANG Q, NI B, et al. Modeling point clouds with self-attention and gumbel subset sampling[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 3323-3332.
[38] ZHANG Z, HUA B S, YEUNG S K. ShellNet: efficient point cloud convolutional neural networks using concentric shells statistics[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 1607-1616.
[39] LEI H, AKHTAR N, MIAN A. Spherical kernel for efficient graph convolution on 3D point clouds[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(10): 3664-3680.
[40] GUO M H, CAI J X, LIU Z N, et al. PCT: point cloud transformer[J]. Computational Visual Media, 2021, 7: 187-199.
[41] TANG L, ZHAN Y, CHEN Z, et al. Contrastive boundary learning for point cloud segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 8489-8499.
[42] WANG L, HUANG Y, HOU Y, et al. Graph attention convolution for point cloud semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 10296-10305.