Research of Deep Learning-Based Semantic Segmentation for 3D Point Cloud

doi:10.3778/j.issn.1002-8331.2107-0142

Abstract

Abstract:

This paper summarizes the methods of deep learning-based semantic segmentation for 3D point cloud. The literature research method is used to describe deep learning-based semantic segmentation methods for 3D point cloud according to the representation of data. It discusses the current situation of domestic and foreign development in recent years, and analyzes the advantages and disadvantages of the current related methods, and prospects the future development trend. Deep learning plays an extremely important role in the research of semantic segmentation technology for point cloud, and promotes the manufacturing, packaging fields and etc to development in the direction of intelligence. According to the advantages and disadvantages of various methods, it is an important research direction to construct a framework model of semantic segmentation combined with 2D-3D for projection, voxel, multi-view and point cloud in the future.

Key words: computer vision, intelligent packaging, deep learning, 3D point cloud, semantic segmentation

摘要：

综述了基于深度学习的三维点云语义分割方法的研究进展。利用文献分析法，按照数据的表现形式对基于深度学习的三维点云语义分割的方法进行阐述。探讨了近些年的国内外发展现状，分析了目前相关方法的优缺点，并展望了未来发展的趋势。深度学习的加入在点云语义分割技术研究上发挥着越来越重要的作用，推动了制造与包装等领域趋向于智能信息化。根据各类方法的优缺点，利用深度学习技术构建出基于投影、体素、多视图以及直接基于点云的2D-3D组合语义分割框架模型是未来的一个重要研究方向。

关键词: 计算机视觉, 智能包装, 深度学习, 三维点云, 语义分割

WANG Tao, WANG Wenju, CAI Yu. Research of Deep Learning-Based Semantic Segmentation for 3D Point Cloud[J]. Computer Engineering and Applications, 2021, 57(23): 18-26.

王涛，王文举，蔡宇. 基于深度学习的三维点云语义分割方法研究[J]. 计算机工程与应用, 2021, 57(23): 18-26.

References

[1] ZHANG R，LI G，WUNDERLICH T，et al.A survey on deep learning-based precise boundary recovery of semantic segmentation for images and point clouds[J].International Journal of Applied Earth Observation and Geoinformation，2021，102：102411.
[2] MOHAMMED M Q，KWEK L C，CHUA S C，et al.Color matching based approach for robotic grasping[C]//2021 International Congress of Advanced Technology and Engineering（ICOTEN），2021：1-8.
[3] ZHANG F，FANG J，WAH B，et al.Deep FusionNet for point cloud semantic segmentation[C]//16th European Conference on Computer Vision，Glasgow，UK，2020：644-663.
[4] MA Y，GUO Y，LIU H，et al.Global context reasoning for semantic segmentation of 3D point clouds[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision，2020：2931-2940.
[5] ZHAO N，CHUA T S，LEE G H.Few-shot 3d point cloud semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：8873-8882.
[6] GUL S，FULALY M S，KHAN M S，et al.Clustering of spatial cues by semantic segmentation for anechoic binaural source separation[J].Applied Acoustics，2021，171：107566.
[7] BENINI S，KHAN K，LEONARDI R，et al.FASSEG：a face semantic segmentation repository for face image analysis[J].Data in Brief，2019，24：103881.
[8] WU Y，RIVENSON Y，ZHANG Y，et al.Extended depth-of-field in holographic imaging using deep-learning-based autofocusing and phase recovery[J].Optica，2018，5（6）：704-710.
[9] ZHANG Y，ZHOU Z，DAVID P，et al.PolarNET：an improved grid representation for online lidar point clouds semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：9601-9610.
[10] MI X，YANG B，CHEN C，et al.Automatic road structure detection and vectorization using MLS point clouds[J].International Archives of the Photogrammetry，Remote Sensing and Spatial Information Sciences，2019，42：1069-1075.
[11] QI C R，SU H，MO K，et al.PointNet：deep learning on point sets for 3D classification and segmentation[C]//30th IEEE Conference on Computer Vision and Pattern Recognition，2017：77-85.
[12] KALOGERAKIS E，HERTZMANN A，SINGH K.Learning 3D mesh segmentation and labeling[J].ACM Transactions on Graphics，2010，29（4）：1-12.
[13] HINTON G E，SALAKHUTDINOV R R.Reducing the dimensionality of data with neural networks[J].Science，2006，313（5786）：504-507.
[14] LECUN Y，BENGIO Y，HINTON G.Deep learning[J].Nature，2015，521（7553）：436-444.
[15] HU Q，YANG B，XIE L，et al.RandLA-Net：efficient semantic segmentation of large-scale point clouds[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：11108-11117.
[16] ARSHAD S，SHAHZAD M，RIAZ Q，et al.DPRNet：deep 3D point based residual network for semantic segmentation and classification of 3D point clouds[J].IEEE Access，2019，7：68892-68904.
[17] ISHIKAWA Y，HACHIUMA R，IENAGA N，et al.Semantic segmentation of 3D point cloud to virtually manipulate real living space[C]//2019 12th Asia Pacific Workshop on Mixed and Augmented Reality（APMAR），2019：1-7.
[18] PENG S，JIANG W，PI H，et al.Deep snake for real-time instance segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：8533-8542.
[19] WU Z，SONG S，KHOSLA A，et al.3D ShapeNets：a deep representation for volumetric shapes[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：1912-1920.
[20] HANDA A，P?TR?UCEAN V，STENT S，et al.SceneNet：an annotated model generator for indoor scene understanding[C]//2016 IEEE International Conference on Robotics and Automation（ICRA），2016：5737-5743.
[21] HUA B S，PHAM Q H，NGUYEN D T，et al.SceneNN：a scene meshes dataset with annotations[C]//2016 Fourth International Conference on 3D Vision（3DV），2016：92-101.
[22] YI L，SHAO L，SAVVA M，et al.Large-scale 3d shape reconstruction and segmentation from shapenet core55[J].arXiv：1710.06104，2017.
[23] ARMENI I，SAX S，ZAMIR A R，et al.Joint 2d-3d-semantic data for indoor scene understanding[J].arXiv：1702.01105，2017.
[24] DAI A，CHANG A X，SAVVA M，et al.ScanNet：richly-annotated 3d reconstructions of indoor scenes[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：5828-5839.
[25] HACKEL T，SAVINOV N，LADICKY L，et al.Semantic-3D.net：a new large-scale point cloud classification benchmark[J].arXiv：1704.03847，2017.
[26] WU B，WAN A，YUE X，et al.SqueezeSeg：convolutional neural nets with recurrent CRF for real-time road-object segmentation from 3d LiDAR point cloud[C]//2018 IEEE International Conference on Robotics and Automation（ICRA），2018：1887-1893.
[27] IANDOLA F N，HAN S，MOSKEWICZ M W，et al.SqueezeNet：AlexNet-level accuracy with 50x fewer parameters and<0.5?MB model size[J].arXiv：1602.07360，2016.
[28] WU B，ZHOU X，ZHAO S，et al.SqueezeSegV2：improved model structure and unsupervised domain adaptation for road-object segmentation from a LiDAR point cloud[C]//2019 International Conference on Robotics and Automation（ICRA），2019：4376-4382.
[29] IMAD M，DOUKHI O，LEE D J.Transfer learning based semantic segmentation for 3D object detection from point cloud[J].Sensors，2021，21（12）：3964.
[30] WANG P S，LIU Y，GUO Y X，et al.O-CNN：octree-based convolutional neural networks for 3d shape analysis[J].ACM Transactions on Graphics，2017，36（4）：1-11.
[31] TCHAPMI L，CHOY C，ARMENI I，et al.SEGCloud：semantic segmentation of 3d point clouds[C]//2017 International Conference on 3D Vision（3DV），2017：537-547.
[32] DAI A，RITCHIE D.ScanComplete：large-scale scene completion and semantic segmentation for 3d scans[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：4578-4587.
[33] LONG J，SHELHAMER E，DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：3431-3440.
[34] LE T，DUAN Y.PointGrid：a deep network for 3d shape understanding[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：9204-9214.
[35] MENG H Y，GAO L，LAI Y K，et al.Vv-net：voxel vae net with group convolutions for point cloud segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：8500-8508.
[36] SU H，MAJI S，KALOGERAKIS E，et al.Multi-view convolutional neural networks for 3d shape recognition[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：945-953.
[37] JIANG J，BAO D，CHEN Z，et al.MLVCNN：multi-loop-view convolutional neural network for 3D shape retrieval[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2019：8513-8520.
[38] TATARCHENKO M，PARK J，KOLTUN V，et al.Tangent convolutions for dense prediction in 3d[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：3887-3896.
[39] YU T，MENG J，YUAN J.Multi-view harmonized bilinear network for 3d object recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：186-194.
[40] LI J，ZHANG X，LI J，et al.Building and optimization of 3D semantic map based on LiDAR and camera fusion[J].Neurocomputing，2020，409：394-407.
[41] QI C R，YI L，SU H，et al.PointNet++：deep hierarchical feature learning on point sets in a metric space[C]//31st Annual Conference on Neural Information Processing Systems，2017：5100-5109.
[42] REN S，HE K，GIRSHICK R，et al.Faster R-CNN：towards real-time object detection with region proposal networks[C]//Advances in Neural Information Processing Systems，2015：91-99.
[43] KLOKOV R，LEMPITSKY V.Escape from cells：deep kd-networks for the recognition of 3d point cloud models[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：863-872.
[44] THOMAS H，QI C R，DESCHAUD J E，et al.KPConv：flexible and deformable convolution for point clouds[C]//Proceedings of the IEEE International Conference on Computer Vision，2019：6411-6420.
[45] ENGELMANN F，KONTOGIANNI T，HERMANS A，et al.Exploring spatial context for 3D semantic segmentation of point clouds[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops，2017：716-724.
[46] ENGELMANN F，BOKELOH M，FATHI A，et al.3D-MPA：multi-proposal aggregation for 3D semantic instance segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：9031-9040.
[47] LANDRIEU L，SIMONOVSKY M.Large-scale point cloud semantic segmentation with superpoint graphs[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：4558-4567.
[48] LANDRIEU L，BOUSSAHA M.Point cloud oversegmentation with graph-structured deep metric learning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2019：7440-7449.
[49] READING C，HARAKEH A，CHAE J，et al.Categorical depth distribution network for monocular 3d object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：8555-8564.
[50] QIU S，ANWAR S，BARNES N.Semantic segmentation for real point cloud scenes via bilateral augmentation and adaptive fusion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：1757-1767.
[51] JIANG M，WU Y，ZHAO T，et al.PointSIFT：a SIFT-like network module for 3d point cloud semantic segmentation[J].arXiv：1807.00652，2018.
[52] LI Y Y，BU R，SUN M C，et al.PointCNN：convolution on x-transformed points[C]//Advances in Neural Information Processing Systems，2018：820-830.
[53] SUN X，LIAN Z，XIAO J.SRINet：learning strictly rotation invariant representations for point cloud classification and segmentation[C]//Proceedings of the 27th ACM International Conference on Multimedia，2019：980-988.
[54] YANG J，ZHANG Q，NI B，et al.Modeling point clouds with self-attention and gumbel subset sampling[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2019：3323-3332.
[55] HU W，ZHAO H，JIANG L，et al.Bidirectional projection network for cross dimension scene understanding[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：14373-14382.
[56] CHEN H，LIU S，CHEN W，et al.Equivariant point network for 3D point cloud analysis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：14514-14523.
[57] MILIOTO A，VIZZO I，BEHLEY J，et al.RangeNet++：fast and accurate LiDAR semantic segmentation[C]//2019 IEEE/RSJ International Conference on Intelligent Robots and Systems（IROS），2019：4213-4220.
[58] GRAHAM B，ENGELCKE M，VAN DER MAATEN L.
3d semantic segmentation with submanifold sparse convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：9224-9232.
[59] DAI A，NIE?NER M.3DMV：joint 3d-multi-view prediction for 3d semantic scene segmentation[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：452-468.
[60] PENG S，JIANG W，PI H，et al.Deep snake for real-time instance segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：8533-8542.
[61] QI L，JIANG L，LIU S，et al.Amodal instance segmentation with kins dataset[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2019：3014-3023.
[62] ACUNA D，KAR A，FIDLER S.Devil is in the edges：learning semantic boundaries from noisy annotations[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2019：11075-11083.