Review of Deep Learning in Point Cloud Classification

doi:10.3778/j.issn.1002-8331.2105-0200

Abstract

Abstract: Deep learning has developed rapidly in the application field of 3D point clouds with the advantages of automatic feature extraction and strong generalization ability. It has gradually become the mainstream research method of point cloud classification. According to the different extraction methods, the existing algorithms are summarized into traditional methods and deep learning algorithms. This paper emphasizes the introduction of representative methods and the latest research based on deep learning, summarizes its basic ideas, advantages and disadvantages. It compares and analyzes the experimental results of the main methods. Finally, it looks forward to the future work and research development of deep learning in the field of point cloud classification.

Key words: image processing, point cloud classification, deep learning, feature extraction, convolutional neural network

摘要： 点云数据被广泛用于多种三维场景，深度学习凭借提取特征自动化、泛化能力强等优势在三维点云的应用领域快速发展，逐渐成为点云分类的主流研究方法。根据提取方式的不同，将现有算法归纳为传统方法以及深度学习算法。着重介绍基于深度学习的代表性方法和最新研究，总结其基本思想以及优缺点，对比分析主要方法的实验结果；展望深度学习在点云分类领域的未来工作以及研究发展方向。

关键词: 图像处理, 点云分类, 深度学习, 特征提取, 卷积神经网络

WANG Wenxi, LI Lelin. Review of Deep Learning in Point Cloud Classification[J]. Computer Engineering and Applications, 2022, 58(1): 26-40.

王文曦, 李乐林. 深度学习在点云分类中的研究综述[J]. 计算机工程与应用, 2022, 58(1): 26-40.

References

[1] MONGUS D，?ALIK B.Computationally efficient method for the generation of a digital terrain model from airborne LiDAR data using connected operators[J].IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing，2013，7（1）：340-351.
[2] ZERMAS D，IZZAT I，PAPANIKOLOPOULOS N.Fast segmentation of 3D point clouds：A paradigm on lidar data for autonomous vehicle applications[C]//2017 IEEE International Conference on Robotics and Automation （ICRA），2017：5067-5073.
[3] 张瑞菊，周欣，赵江洪，等.一种古建筑点云数据的语义分割算法[J].武汉大学学报（信息科学版），2020，45（5）：753-759.
ZHANG R J，ZHOU X，ZHAO J H，et al.A semantic segmentation algorithm for point cloud data of an-cient buildings[J].Journal of Wuhan University（Information Science Edition），2020，45（5）：753-759.
[4] 范雯，何鄂龙，李天琪，等.融合空谱特征的车载LiDAR点云道路标识线提取[J].测绘通报，2018（8）：97-101.
FAN W，HE E L，LI T Q，et al.Road marking line extraction of vehicle LiDAR point cloud combining spatial spectrum features[J].Bulletin of Surveying and Mapping，2018（8）：97-101.
[5] 张继贤，林祥国，梁欣廉.点云信息提取研究进展和展望[J].测绘学报，2017，46（10）：1460-1469.
ZHANG J X，LIN X G，LIANG X L.Research progress and prospect of point cloud information extraction[J].Acta Geodaetoca et Cartgraphica Sinica，2017，46（10）：1460-1469.
[6] CHEN C，LI X，BELKACEM A N，et al.The mixed kernel function SVM-based point cloud classification[J].International Journal of Precision Engineering and Manufacturing，2019，20（5）：737-747.
[7] ZHANG J，LIN X，NING X.SVM-based classification of segmented airborne LiDAR point clouds in urban areas[J].Remote Sensing，2013，5（8）：3749-3775.
[8] NI H，LIN X，ZHANG J.Classification of ALS point cloud with improved point cloud segmentation and random forests[J].Remote Sensing，2017，9（3）：288.
[9] WEINMANN M，JUTZI B，HINZ S，et al.Semantic point cloud interpretation based on optimal neighborhoods，relevant features and efficient classifiers[J].ISPRS Journal of Photogrammetry and Remote Sensing，2015，105：286-304.
[10] 孙杰，赖祖龙.利用随机森林的城区机载LiDAR数据特征选择与分类[J].武汉大学学报（信息科学版），2014，39（11）：1310-1313.
SUN J，LAI Z L.Feature selection and classification of urban airborne LiDAR data using random forest[J].Journal of Wuhan University（Information Science Edition），2014，39（11）：1310-1313.
[11] 郭波，黄先锋，张帆，等.顾及空间上下文关系的JointBoost点云分类及特征降维[J].测绘学报，2013，42（5）：715-721.
GUO B，HUANG X F，ZHANG F，et al.JointBoost point cloud classification and feature dimensionality reduction considering spatial context[J].Acta Geodaetoca et Cartgraphica Sinica，2013，42（5）：715-721.
[12] NIEMEYER J，ROTTENSTEINER F，SOERGEL U.Contextual classification of lidar data and building object detection in urban areas[J].ISPRS Journal of Photogrammetry and Remote Sensing，2014，87：152-165.
[13] MUNOZ D，BAGNELL J A，VANDAPEL N，et al.Contextual classification with functional maxmargin Markov networks[C]//2009 IEEE Conference on Computer Vision and Pattern Recognition，2009：975-982.
[14] SHAPOVALOV R，VELIZHEV E，BARINOVA O.Nonassociative Markov networks for 3D point cloud classification[C]//International Archives of the Photogrammetry，Remote Sensing and Spatial Information Sciences，2010.
[15] 徐浩钧.基于深度学习的点云分类技术研究[D].银川：北方民族大学，2020.
XU H J.Research on point cloud classification technology based on deep learning[D].Yinchuan：Northern University for Nationalities，2020.
[16] MATURANA D，SCHERER S.Voxnet：A 3D convolutional neural network for real-time object recognition[C]//2015 IEEE/RSJ International Conference on Intelligent Robots and Systems（IROS），2015：922-928.
[17] WU Z，SONG S，KHOSLA A，et al.3D shapenets：A deep representation for volumetric shapes[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：1912-1920.
[18] COHEN T S，GEIGER M，K?HLER J，et al.Spherical CNNs[J].arXiv：1801.10130，2018.
[19] YOU Y，LOU Y，LIU Q，et al.Pointwise rotation-invariant network with adaptive sampling and 3D spherical voxel convolution[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2020：12717-12724.
[20] RIEGLER G，OSMAN ULUSOY A，GEIGER A.Octnet：Learning deep 3D representations at high resolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：3577-3586.
[21] WANG P S，LIU Y，GUO Y X，et al.O-CNN：Octree-based convolutional neural networks for 3D shape analysis[J].ACM Transactions on Graphics（TOG），2017，36（4）：1-11.
[22] KLOKOV R，LEMPITSKY V.Escape from cells：Deep KD-networks for the recognition of 3D point cloud models[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：863-872.
[23] HANOCKA R，HERTZ A，FISH N，et al.MeshCNN：A network with an edge[J].ACM Transactions on Graphics（TOG），2019，38（4）：1-12.
[24] SU H，MAJI S，KALOGERAKIS E，et al.Multi-view convolutional neural networks for 3D shape recognition[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：945-953.
[25] QI C R，SU H，NIE?NER M，et al.Volumetric and multi-view CNNs for object classification on 3D data[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：5648-5656.
[26] KALOGERAKIS E，AVERKIOU M，MAJI S，et al.3D shape segmentation with projective convo-lutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：3779-3788.
[27] FENG Y，ZHANG Z，ZHAO X，et al.GVCNN：Group-view convolutional neural networks for 3D shape recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：264-272.
[28] MA C，GUO Y，YANG J，et al.Learning multi-view representation with LSTM for 3D shape recognition and retrieval[J].IEEE Transactions on Multimedia，2018，21（5）：1169-1182.
[29] YU T，MENG J，YUAN J.Multi-view harmonized bilinear network for 3D object recogni-tion[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：186-194.
[30] QI C R，SU H，MO K，et al.Pointnet：Deep learning on point sets for 3D classification and segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：652-660.
[31] QI C R，YI L，SU H，et al.Pointnet++：Deep hierarchical feature learning on point sets in a metric space[J].arXiv：1706.02413，2017.
[32] JOSEPH-RIVLIN M，ZVIRIN A，KIMMEL R.Moment：Flavor the moments in learning to classify shapes[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops，2019.
[33] LI J，CHEN B M，LEE G H.So-net：Self-organizing network for point cloud analysis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：9397-9406.
[34] DUAN Y，ZHENG Y，LU J，et al.Structural relational reasoning of point clouds[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：949-958.
[35] ZHAO H，JIANG L，FU C W，et al.Pointweb：Enhancing local neighborhood features for point cloud processing[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：5565-5573.
[36] YAN X，ZHENG C，LI Z，et al.Pointasnl：Robust point clouds processing using nonlocal neural networks with adaptive sampling[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：5589-5598.
[37] PROKUDIN S，LASSNER C，ROMERO J.Efficient learning on point clouds with basis point sets[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：4332-4341.
[38] YOUSEFHUSSIEN M，KELBE D J，IENTILUCCI E J，et al.A multi-scale fully convolutional network for semantic labeling of 3D point clouds[J].ISPRS Journal of Photogrammetry and Remote Sensing，2018，143：191-204.
[39] WANG Z，ZHANG L，ZHANG L，et al.A deep neural network with spatial pooling（DNNSP） for 3D point cloud classification[J].IEEE Transactions on Geoscience and Remote Sensing，2018，56（8）：4594-4604.
[40] KOMARICHEV A，ZHONG Z，HUA J.A-CNN：Annularly convolutional neural networks on point clouds[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：7421-7430.
[41] LI Y，BU R，SUN M，et al.Pointcnn：Convolution on [x]-transformed points[J].Advances in Neural Information Processing Systems，2018，31：820-830.
[42] LIU Y，FAN B，XIANG S，et al.Relation-shape convolutional neural network for point cloud analysis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：8895-8904.
[43] WU W，QI Z，FUXIN L.Pointconv：Deep convolutional networks on 3D point clouds[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：9621-9630.
[44] THOMAS H，QI C R，DESCHAUD J E，et al.Kpconv：Flexible and deformable convolution for point clouds[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：6411-6420.
[45] BOULCH A.ConvPoint：Continuous convolutions for point cloud processing[J].Computers & Graphics，2020，88：24-34.
[46] WEN C，YANG L，LI X，et al.Directionally constrained fully convolutional neural network for airborne LiDAR point cloud classification[J].ISPRS Journal of Photogrammetry and Remote Sensing，2020，162：50-62.
[47] LI X，WANG L，WANG M，et al.DANCE-NET：Density-aware convolution networks with context encoding for airborne LiDAR point cloud classification[J].ISPRS Journal of Photo-grammetry and Remote Sensing，2020，166：128-139.
[48] SIMONOVSKY M，KOMODAKIS N.Dynamic edge-conditioned filters in convolutional neural networks on graphs[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：3693-3702.
[49] SHEN Y，FENG C，YANG Y，et al.Mining point cloud local structures by kernel correlation and graph pooling[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：4548-4557.
[50] WANG Y，SUN Y，LIU Z，et al.Dynamic graph cnn for learning on point clouds[J].ACM Transactions on Graphics，2019，38（5）：1-12.
[51] ZHANG K，HAO M，WANG J，et al.Linked dynamic graph CNN：Learning on point cloud via linking hierarchical features[J].arXiv：1904.10014，2019.
[52] SHI W，RAJKUMAR R.Point-GNN：Graph neural network for 3D object detection in a point cloud[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：1711-1719.
[53] XU Q，SUN X，WU C Y，et al.Grid-GCN for fast and scalable point cloud learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：5661-5670.
[54] LI R，ZHANG Y，NIU D，et al.PointVGG：Graph convolutional network with progressive aggregating features on point clouds[J].Neurocomputing，2021，429：187-198.
[55] WANG W，YOU Y，LIU W，et al.Point cloud classification with deep normalized Reeb graph convolution[J].Image and Vision Computing，2021，106：104092.
[56] YANG J，ZHANG Q，NI B，et al.Modeling point clouds with self-attention and gumbel subset sampling[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：3323-3332.
[57] CHEN C，FRAGONARA L Z，TSOURDOS A.GAPointNet：Graph attention based point neural network for exploiting local feature of point cloud[J].Neurocomputing，2021，438：122-132.
[58] WANG L，HUANG Y，HOU Y，et al.Graph attention convolution for point cloud semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：10296-10305.
[59] GUO M H，CAI J X，LIU Z N，et al.PCT：Point cloud transformer[J].arXiv：2012.09688，2020.
[60] WEN C，LI X，YAO X，et al.Airborne LiDAR point cloud classification with global-local graph attention convolution neural network[J].ISPRS Journal of Photogrammetry and Remote Sensing，2021，173：181-194.
[61] GUO Y L，WANG H，HU Q，et al.Deep learning for 3D point clouds：A survey[J].arXiv：1912.12033，2019.
[62] XIE Y，TIAN J，ZHU X X.Linking points with labels in 3D：A review of point cloud semantic segmentation[J].IEEE Geoscience and Remote Sensing Magazine，2020，8（4）：38-59.
[63] 文沛，程英蕾，余旺盛.基于深度学习的点云分类方法综述[J].激光与光电子学进展，2021，58（16）：1-36.
WEN P，CHENG Y L，YU W S.A survey of point cloud classification methods based on deep learning[J].Advances in Laser and Optoelectronics，2021，58（16）：1-36.
[64] DAI A，CHANG A X，SAVVA M，et al.Scannet：Richly-annotated 3D reconstructions of indoor scenes[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：5828-5839.
[65] ROTTENSTEINER F，SOHN G，JUNG J，et al.The ISPRS benchmark on urban object classification and 3D building reconstruction[J].ISPRS Annals of the Photogrammetry，Remote Sensing and Spatial Information Sciences，2012，1（1）：293-298.
[66] LE SAUX B，YOKOYA N，H?NSCH R，et al.2019 IEEE grss data fusion contest：Large-scale semantic 3D reconstruction[J].IEEE Geoscience and Remote Sensing Magazine（GRSM），2019，7（4）：33-36.
[67] LECUN Y，BENGIO Y，HINTON G.Deep learning[J].Nature，2015，521：436-444.
[68] 景庄伟，管海燕，臧玉府，等.基于深度学习的点云语义分割研究综述[J].计算机科学与探索，2021，15（1）：1-26.
JING Z W，GUAN H Y，ZANG Y F，et al.Survey of point cloud semantic segmentation based on deep learning[J].Journal of Frontiers of Computer Science and Technology，2021，15（1）：1-26.
[69] GORI M，MONFARDINI G，SCARSELLI F.A new model for learning in graph domains[C]//Proceedings of 2005 IEEE International Joint Conference on Neural Networks，2005：729-734.
[70] SCARSELLI F，GORI M，TSOI A C，et al.The graph neural network model[J].IEEE Transactions on Neural Networks，2008，20（1）：61-80.
[71] BRUNA J，ZAREMBA W，SZLAM A，et al.Spectral networks and locally connected networks on graphs[J].arXiv：1312.6203，2013.
[72] KIPF T N，WELLING M.Semi-supervised classification with graph convolutional networks[J].arXiv：1609.02907，2016.
[73] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems，2017：5998-6008.
[74] HU J，SHEN L，SUN G.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：7132-7141.
[75] WANG Q，WU B，ZHU P，et al.ECA-Net：Efficient channel attention for deep convolutional neural networks[C]//Proceedings of CVF Conference on Computer Vision and Pattern Recognition（CVPR），2020.