Survey on Deep-Learning-Based Point Cloud Semantic Segmentation

doi:10.3778/j.issn.1002-8331.2408-0033

Abstract

Abstract: Point cloud semantic segmentation assigns semantic labels to each point in the point cloud to achieve the segmentation of different objects in the scene, which is the foundation for scene understanding. In recent years, with the development of deep learning technology, the combination of deep learning and point cloud semantic segmentation methods has improved the processing efficiency and segmentation accuracy, demonstrating excellent performance, and has been widely used in many fields such as transportation, medicine, architectural design, virtual reality, etc. Based on the review of the development history of point cloud semantic segmentation, this paper classifies and summarizes the existing research, analyzes related datasets and evaluation metrics, and compares the performance of existing methods. Finally, the paper highlights the deficiencies of existing research and looks forward to the future development directions.

Key words: deep learning, point cloud, semantic segmentation, semantic labels, computer vision

摘要： 点云语义分割将点云中每个点赋予语义标签，实现对场景中不同物体的分割，是场景理解的基础。近年来，随着深度学习技术的发展，将深度学习与点云语义分割方法相结合，提升了点云语义分割的处理效率和分割精度，展现出卓越的性能，被广泛应用于交通、医学、建筑设计、虚拟现实等众多领域。在回顾点云语义分割发展历程的基础上，对已有研究进行分类综述，然后分析相关数据集和评价指标，对比已有方法的性能。最后，总结现有研究的不足，并展望未来发展方向。

关键词: 深度学习, 点云, 语义分割, 语义标签, 计算机视觉

CAI Ziyue, YUAN Zhenyue, PANG Mingyong. Survey on Deep-Learning-Based Point Cloud Semantic Segmentation[J]. Computer Engineering and Applications, 2025, 61(11): 22-30.

蔡子悦, 袁振岳, 庞明勇. 深度学习的点云语义分割方法综述[J]. 计算机工程与应用, 2025, 61(11): 22-30.

References

[1] 秦飞巍, 沈希乐, 彭勇, 等. 无人驾驶中的场景实时语义分割方法[J]. 计算机辅助设计与图形学学报, 2021, 33(7): 1026-1037.
QIN F W, SHEN X Y, PENG Y, et al. A real-time semantic segmentation approach for autonomous driving scenes[J]. Journal of Computer-Aided Design & Computer Graphics, 2021, 33(7): 1026-1037.
[2] JIN Y H, HWANG I T, LEE W H. A mobile augmented reality system for the real-time visualization of pipes in point cloud data with a depth sensor[J]. Electronics, 2020, 9(5): 836.
[3] BI X. Environmental perception technology for unmanned systems[M]. Cham: Springer Nature Singapore, 2021: 17-65.
[4] CAI Y M, LONG Y Q, HAN Z G, et al. Swin Unet3D: a three-dimensional medical image segmentation network combining vision transformer and convolution[J]. BMC Medical Informatics and Decision Making, 2023, 23(1): 33-46.
[5] ZHANG J Y, ZHAO X L, CHEN Z, et al. A review of deep learning-based semantic segmentation for point cloud[J]. IEEE Access, 2019, 7: 179118-179133.
[6] ZHANG R, WU Y C, JIN W, et al. Deep-learning-based point cloud semantic segmentation: a survey[J]. Electronics, 2023, 12(17): 3642-3667.
[7] 张佳颖, 赵晓丽, 陈正. 基于深度学习的点云语义分割综述[J]. 激光与光电子学进展, 2020, 57(4): 040002.
ZHANG J Y, ZHAO X L, CHEN Z. Review of semantic segmentation of point cloud based on deep learning[J]. Laser & Optoelectronics Progress, 2020, 57(4): 040002.
[8] 王涛, 王文举, 蔡宇. 基于深度学习的三维点云语义分割方法研究[J]. 计算机工程与应用, 2021, 57(23): 18-26.
WANG T, WANG W J, CAI Y. Research of deep learning-based semantic segmentation for 3D point cloud[J]. Computer Engineering and Applications, 2021, 57(23): 18-26.
[9] 景庄伟, 管海燕, 臧玉府, 等. 基于深度学习的点云语义分割研究综述[J]. 计算机科学与探索, 2021, 15(1): 1-26.
JING Z W, GUAN H Y, ZANG Y F, et al. Survey of point cloud semantic segmentation based on deep learning[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(1): 1-26.
[10] SU H, MAJI S, KALOGERAKIS E, et al. Multi-view convolutional neural networks for 3D shape recognition[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway: IEEE, 2015: 945-953.
[11] FENG Y F, ZHANG Z Z, ZHAO X B, et al. GVCNN: group-view convolutional neural networks for 3D shape recognition[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 264-272.
[12] MATURANA D, SCHERER S. VoxNet: a 3D convolutional neural network for real-time object recognition[C]//Proceedings of the 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway: IEEE, 2015: 922-928.
[13] RIEGLER G, ULUSOY A O, GEIGER A. OctNet: learning deep 3D representations at high resolutions[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 6620-6629.
[14] QI C R, SU H, MO K, et al. PointNet: deep learning on point sets for 3D classification and segmentation[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 77-85.
[15] JIANG M Y, WU Y R, ZHAO T Q, et al. PointSIFT: a SIFT-like network module for 3D point cloud semantic segmentation[J]. arXiv:1807.00652, 2018.
[16] ZHAO H S, JIANG L, FU C W, et al. PointWeb: enhancing local neighborhood features for point cloud processing[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 5565-5573.
[17] YE X Q, LI J M, HUANG H X, et al. 3D recurrent neural networks with context fusion for point cloud semantic segmentation[C]//Proceedings of the European Conference on Computer Vision. Cham: Springer International Publishing, 2018: 415-430.
[18] HUANG Q G, WANG W Y, NEUMANN U. Recurrent slice networks for 3D segmentation of point clouds[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 2626-2635.
[19] WANG Y, SUN Y B, LIU Z W, et al. Dynamic graph CNN for learning on point clouds[J]. ACM Transactions on Graphics, 2019, 38(5): 1-12.
[20] LI R K, ZHANG Y M, NIU D M, et al. PointVGG: graph convolutional network with progressive aggregating features on point clouds[J]. Neurocomputing, 2021, 429: 187-198.
[21] 张华, 徐瑞政, 郑南山, 等. 融合颜色信息和多尺度几何特征的点云语义分割方法[J]. 地球信息科学学报, 2024, 26(6): 1562-1575.
ZHANG H, XU R Z, ZHENG N S, et al. Integrating color information and multi-scale geometric features for point cloud semantic segmentation[J]. Journal of Geo-Information Science, 2024, 26(6): 1562-1575.
[22] ZHANG F H, FANG J, WAH B, et al. Deep FusionNet for point cloud semantic segmentation[C]//Proceedings of the 16th European Conference on Computer Vision. Cham: Springer International Publishing, 2020: 644-663.
[23] YE M S, XU S J, CAO T Y, et al. DRINet: a dual-representation iterative learning network for point cloud segmentation[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021: 7427-7436.
[24] LI X Y, ZHANG G, JIANG T, et al. PRNet: point-range fusion network for real-time LiDAR semantic segmentation[C]//Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, 2022: 1116-1122.
[25] XU J Y, ZHANG R X, DOU J, et al. RPVNet: a deep and efficient range-point-voxel fusion network for LiDAR point cloud segmentation[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021: 16004-16013.
[26] YOU H X, FENG Y F, ZHAO X B, et al. PVRNet: point-view relation neural network for 3D shape recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2019: 9119-9126.
[27] ROBERT D, VALLET B, LANDRIEU L. Learning multi-view aggregation in the wild for large-scale 3D semantic segmentation[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 5565-5574.
[28] WANG C, PELILLO M, SIDDIQI K. Dominant set clustering and pooling for multi-view 3D object recognition[J]. arXiv:1906.01592, 2019.
[29] TCHAPMI L, CHOY C, ARMENI I, et al. SEGCloud: semantic segmentation of 3D point clouds[C]//Proceedings of the 2017 International Conference on 3D Vision. Piscataway: IEEE, 2017: 537-547.
[30] WU Z R, SONG S R, KHOSLA A, et al. 3D ShapeNets: a deep representation for volumetric shapes[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2015: 1912-1920.
[31] WANG P S, LIU Y, GUO Y X, et al. O-CNN[J]. ACM Transactions on Graphics, 2017, 36(4): 1-11.
[32] MENG H Y, GAO L, LAI Y K, et al. VV-Net: voxel VAE net with group convolutions for point cloud segmentation[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2019: 8499-8507.
[33] ALEXANDRU ROSU R, SCHüTT P, QUENZEL J, et al. LatticeNet: fast point cloud segmentation using permutohedral lattices[J].arXiv:1912.05905, 2019.
[34] QI C R, YI L, SU H, et al. PointNet++: deep hierarchical feature learning on point sets in a metric space[J]. arXiv: 1706. 02413, 2017.
[35] ZHANG Z Y, HUA B S, YEUNG S K. ShellNet: efficient point cloud convolutional neural networks using concentric shells statistics[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2019: 1607-1616.
[36] HU Q Y, YANG B, XIE L H, et al. RandLA-Net: efficient semantic segmentation of large-scale point clouds[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 11105-11114.
[37] MA X, QIN C, YOU H X, et al. Rethinking network design and local geometry in point cloud: a simple residual MLP framework[J]. arXiv:2202:07123, 2022.
[38] ZHAN L X, LI W, MIN W D. FA-ResNet: feature affine residual network for large-scale point cloud segmentation[J]. International Journal of Applied Earth Observation and Geoinformation, 2023, 118: 103259.
[39] ZHAO Z, LIU M, RAMANI K. Dynamic aggregation network for semantic scene segmentation[J]. arXiv:1907.12022, 2019.
[40] LIU F Y, LI S P, ZHANG L Q, et al. 3DCNN-DQN-RNN: a deep reinforcement learning framework for semantic parsing of large-scale 3D point clouds[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway: IEEE, 2017: 5679-5688.
[41] SIMONOVSKY M, KOMODAKIS N. Dynamic edge-conditioned filters in convolutional neural networks on graphs[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 29-38.
[42] LANDRIEU L, SIMONOVSKY M. Large-scale point cloud semantic segmentation with superpoint graphs[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 4558-4567.
[43] LEI H, AKHTAR N, MIAN A. Spherical kernel for efficient graph convolution on 3D point clouds[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(10): 3664-3680.
[44] LU Q, CHEN C, XIE W J, et al. PointNGCNN: deep convolutional networks on 3D point clouds with neighborhood graph filters[J]. Computers & Graphics, 2020, 86: 42-51.
[45] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[[J]. arXiv:1409. 1556, 2014.
[46] ZHANG K G, HAO M, WANG J, et al. Linked dynamic graph CNN: learning on point cloud via linking hierarchical features[J]. arXiv:1904.10014, 2019.
[47] WANG L, HUANG Y C, HOU Y L, et al. Graph attention convolution for point cloud semantic segmentation[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 10296-10305.
[48] SHEN Y R, FENG C, YANG Y Q, et al. Mining point cloud local structures by kernel correlation and graph pooling[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 4548-4557.
[49] GENG Y X, WANG Z P, JIA L M, et al. 3DGraphSeg: a unified graph representation- based point cloud segmentation framework for full-range high-speed railway environments[J]. IEEE Transactions on Industrial Informatics, 2023, 19(12): 11430-11443.
[50] ZHANG R, LI G Y, WIEDEMANN W, et al. KdO-Net: towards improving the efficiency of deep convolutional neural networks applied in the 3D pairwise point feature matching[J]. Remote Sensing, 2022, 14(12): 2883.
[51] ZHANG N, PAN Z Y, LI T H, et al. Improving graph representation for point cloud segmentation via attentive filtering[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2023: 1244-1254.
[52] YUE G W, XIAO R, ZHAO Z Y, et al. AF-GCN: attribute-fusing graph convolution network for recommendation[J]. IEEE Transactions on Big Data, 2023, 9(2): 597-607.
[53] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[J]. arXiv.1706.03762, 2017.
[54] YANG J C, ZHANG Q, NI B B, et al. Modeling point clouds with self-attention and gumbel subset sampling[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 3323-3332.
[55] GUO M H, CAI J X, LIU Z N, et al. PCT: point cloud transformer[J]. Computational Visual Media, 2021, 7(2): 187-199.
[56] ZHAO H S, JIANG L, JIA J Y, et al. Point transformer[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021: 16239-16248.
[57] ENGEL N, BELAGIANNIS V, DIETMAYER K. Point transformer[J]. IEEE Access, 2021, 9: 134826-134840.
[58] ZHONG Q, HAN X F. Point cloud learning with transformer[J]. arXiv:2104.13636, 2021.
[59] HAN X F, JIN Y F, CHENG H X, et al. Dual transformer for point cloud analysis[J]. IEEE Transactions on Multimedia, 2022, 25: 5638-5648.
[60] LAI X, LIU J H, JIANG L, et al. Stratified transformer for 3D point cloud segmentation[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 8490-8499.
[61] ZHOU J J, XIONG Y P, CHIU C, et al. SAT: size-aware transformer for 3D point cloud semantic segmentation[J]. arXiv:2301.06869, 2023.
[62] ZHOU W, ZHAO Y H, XIAO Y, et al. TNPC: transformer-based network for point cloud classification[J]. Expert Systems with Applications, 2024, 239: 122438.
[63] JIANG C R, HUANG K Z, WU J W, et al. PointGS: bridging and fusing geometric and semantic space for 3D point cloud analysis[J]. Information Fusion, 2023, 91: 316-326.
[64] WU J W, SUN M J, JIANG C R, et al. Context-based local-global fusion network for 3D point cloud classification and segmentation[J]. Expert Systems with Applications, 2024, 251: 124023.
[65] CHENG H Z, ZHU J H, LU J, et al. EDGCNet: joint dynamic hyperbolic graph convolution and dual squeeze-and-attention for 3D point cloud segmentation[J]. Expert Systems with Applications, 2024, 237: 121551.