Multi-Scale Hierarchical Point Clouds Recognition Network Based on Weight Index of Point Density

doi:10.3778/j.issn.1002-8331.2011-0388

Abstract

Abstract: Unlike dense and regularly distributed 2D raster images, 3D point clouds are irregular and disordered, thus applying convolution on them can be difficult. A method for direct convolution of original 3D point clouds is proposed. This method uses Gaussian kernel density estimation and multi-layer perceptron（MLP） network to learn the density function. Besides, the density scale of the learning point and the relative position of the point are fitted by the weight function of the approximated MLP network to obtain the weight value of each point in the local area. The whole convolution kernel can be seen as a nonlinear function of 3D local coordinates composed of the weight function and the density function, which can be used for translation invariant and permutation invariant convolution of any set of points in 3D space. In addition, multi-scale sampling grouping and normal features are employed to achieve the best effect of this network. In the classification experiment of point cloud datasets of ModelNet40 and ModelNet10, the accuracy of the network is 92.8% and 94.7% respectively, which acquires better performance than baseline. The image datasets of CIFAR-10 and MNIST are converted into point clouds to test, experiments on aforementioned datasets show that the performance of the network in 2D images is basically equivalent to that of the traditional 2D convolutional network.

Key words: image processing, 3D point cloud, kernel density estimation, convolution computation, multi-scale features, normal features

摘要： 与密集且规则分布的2D栅格状图像不同，3D点云是不规则且无序的，对其进行卷积可能会存在一定的困难，因此，提出了一种针对原始3D点云的卷积运算。该方法使用高斯核密度估计和多层感知器（multi-layer perceptron，MLP）网络来学习密度函数，将学习到的点的密度尺度结合点的相对位置，通过由MLP网络近似的权重函数之中，得到局域中每个点的权重值。整个卷积核可视为由权重函数和密度函数组成的3D点局域坐标的非线性函数，可用于对3D空间中任意点集进行平移不变和置换不变的卷积，并融合多尺度采样分组和法向特征使网络达到最佳效果。在ModelNet40和ModelNet10数据集的分类实验中，该网络分别取得了92.8%和94.7%的准确率，均高于所对比的同类方法的性能水平。将CIFAR-10和MNIST图像数据集转为点云并进行测试，结果表明网络在2D图像中的性能基本等效于传统2D卷积网络。

关键词: 图像处理, 3D点云, 核密度估计, 卷积运算, 多尺度特征, 法向特征

QIN Xinyu, HAN Shuai, SHEN Xueli, YANG Ying. Multi-Scale Hierarchical Point Clouds Recognition Network Based on Weight Index of Point Density[J]. Computer Engineering and Applications, 2022, 58(13): 217-226.

秦鑫宇, 韩帅, 沈学利, 杨莹. 基于点密度加权的多尺度分层点云识别网络[J]. 计算机工程与应用, 2022, 58(13): 217-226.

References

[1] GUO Y，WANG H，HU Q，et al.Deep learning for 3D point clouds：A survey[EB/OL].（2020-06-23）[2020-11-25].http：//arxiv.org/abs/1912.12033.
[2] 张婧，周明全，耿国华.点云数据在深度学习中表示方法的研究[J].计算机工程与应用，2020，56（18）：171-176.
ZHANG J，ZHOU M Q，GENG G H.Research on point cloud data representation method in deep learning[J].Computer Engineering and Applications，2020，56（18）：171-176.
[3] MATURANA D，SCHERER S.VoxNet：A 3D convolutional neural network for real-time object recognition[C]//Proceedings of International Conference on Intelligent Robots and Systems，2015：922-928.
[4] SU H，MAJI S，KALOGERAKIS E，et al.Multi-view convolutional neural networks for 3D shape recognition[C]//Proceedings of the IEEE International Conference on Computer Vision，Dec 13-16，2015：945-953.
[5] KALOGERAKIS E，AVERKIOU M，MAJI S，et al.3D shape segmentation with projective convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Jul 21-26，2017：3779-3788.
[6] 杨书娟，张珂殊，邵永社.基于多尺度自适应特征的机载LiDAR点云分类[J].光学学报，2019，39（2）：0228001.
YANG S J，ZHANG K S，SHAO Y S.Classification of airborne LiDAR point cloud data based on multiscale adaptive features[J].Acta Agronomica Sinica，2019，39（2）：0228001.
[7] 卢晓艺，云挺，薛联凤，等.基于树木激光点云的有效特征抽取与识别方法[J].光学学报，2019，46（5）：0510002.
LU X Y，YUN T，XUE L F，et al.Effective feature extraction and identification method based on tree laser point cloud[J].Acta Agronomica Sinica，2019，46（5）：0510002
[8] 俞斌，董晨，刘延华，等.基于深度学习的点云分割方法综述[J].计算机工程与应用，2020，56（1）：38-45.
YU B，DONG C，LIU Y H，et al.Deep learning based point cloud segmentation：A survey[J].Computer Engineering and Applications，2020，56（1）：38-45.
[9] QI C R，SU H，MO K，et al.PointNet：Deep learning on point sets for 3D classification and segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Jul 21-26，2017：652-660.
[10] 马京晖，潘巍，王茹.基于K-means聚类的三维点云分类[J].计算机工程与应用，2020，56（17）：181-186.
MA J H，PAN W，WANG R.3D point cloud classification based on K-means clustering[J].Computer Engineering and Applications，2020，56（17）：181-186.
[11] 赵中阳，程英蕾，释小松，等.基于多尺度特征和PointNet的LiDAR点云地物分类方法[J].激光与光电子学进展，2019，56（5）：052804.
ZHAO Z Y，CHENG Y L，SHI X S，et al.Terrain classif ication of LiDAR point cloud based on multi-scale features and PointNet[J].Laser & Optoelectronics Progress，2019，56（5）：052804.
[12] 王宏涛，雷相达，赵宗泽.融合光谱信息的机载LiDAR点云三维深度学习分类方法[J].激光与光电子学进展，2020，57（12）：122802.
WANG H T，LEI X D，ZHAO Z Z.3D deep learning classification method for airborne LiDAR point clouds fusing spectral information[J].Laser & Optoelectronics Progress，2020，57（12）：122802.
[13] QI C R，YI L，SU H，et al.Pointnet++：Deep hierarchical feature learning on point sets in a metric space[C]//Advances in Neural Information Processing Systems，Dec 4-9，2017：5099-5108.
[14] JIA X，DE BRABANDERE B，TUYTELAARS T，et al.Dynamic filter networks[C]//Advances in Neural Information Processing Systems，Dec 5-10，2016：667-675.
[15] SIMONOVSKY M，KOMODAKIS N.Dynamic edge-conditioned filters in convolutional neural networks on graphs[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Jul 21-26，2017：3693-3702.
[16] WANG S，SUO S，MA W C，et al.Deep parametric continuous convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Jun 19-21，2018：2589-2597.
[17] CHIU S T.A comparative review of bandwidth selection for kernel density estimation[J].Statistica Sinica，1996，6（6）：126-145.
[18] GONZALEZ R C，WOODS R E.数字图像处理[M].2版.北京：电子工业出版社，2005：545-548.
GONZALEZ R C，WOODS R E.Digital image processing[M].2nd ed.Beijing：Electronic Industry Press，2005：545-548.
[19] YU T，LU L，LI J.A weight-bounded importance sampling method for variance reduction[J].International Journal for Uncertainty Quantification，2019，9（3）：311-319.
[20] ELDAR Y，LINDENBAUM M，PORAT M，et al.The farthest point strategy for progressive image sampling[J].IEEE Transactions on Image Processing，1997，6（9）：1305-1315.
[21] Enthought.scipy.spatial.KDTree.query_ball_point[EB/OL].（2014-05-11）[2020-11-25].https：//docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.sptial.KDTree.query_ball_
point.html#scipy.spatial.KDTree.query_ball_point.
[22] CAI J R，KUANG L Q，HAN X.Multi-scale feature point extraction algorithm based on scattered point cloud[J].Computer Engineering and Design，2016，37（12）：3255-3259.
[23] WU Z，SONG S，KHOSLA A，et al.3D shapenets：A deep representation for volumetric shapes[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Jun 8-12，2015：1912-1920.
[24] BAI S，BAI X，ZHOU Z，et al.Gift：A real-time and scalable 3d shape search engine[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Jun 26-Jul 7，2016：5023-5032.
[25] LIU S，GILES L，ORORBIA A.Learning a hierarchical latent-variable model of 3D shapes[C]//Proceedings of the 2018 International Conference on 3D Vision，Sep 5-8，2018：542-551.
[26] MA C，GUO Y，LEI Y，et al.Binary volumetric convolutional neural networks for 3D object recognition[J].IEEE Transactions on Instrumentation and Measurement，2018，68（1）：38-48.
[27] ZHI S，LIU Y，LI X，et al.Toward real-time 3D object recognition：A lightweight volumetric CNN framework using multitask learning[J].Computers & Graphics，2018，71：199-207.
[28] HEGDE V，ZADEH R.FusionNet：3D object classification using multiple data representations[EB/OL].（2016-09-27）[2020-07-17].https：//arxiv.org/abs/1607.05695.
[29] SHEN Y，FENG C，YANG Y，et al.Mining point cloud local structures by kernel correlation and graph pooling[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Jun 19-21，2018：4548-4557.
[30] LIU Y，FAN B，XIANG S，et al.Relation-shape convolutional neural network for point cloud analysis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Jun 16-20，2019：8895-8904.
[31] SIMARD P Y，STEINKRAUS D，PLATT J C.Best practices for convolutional neural networks applied to visual document analysis[C]//Proceedings of the 7th International Conference on Document Analysis and Recognition，2003：958.
[32] LECUN Y，BOTTOU L，BENGIO Y，et al.Gradient-based learning applied to document recognition[J].Proceedings of the IEEE，1998，86（11）：2278-2324.
[33] LIN M，CHEN Q，YAN S.Network in network[EB/OL].（2014-03-04）[2020-07-17].https：//arxiv.org/abs/1312.4400.
[34] KRIZHEVSKY A，SUTSKEVER I，HINTON G E.Image-net classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems，Dec 3-8，2012：1097-1105.
[35] XU Y，FAN T，XU M，et al.SpiderCNN：Deep learning on point sets with parameterized convolutional filters[C]//Proceedings of the European Conference on Computer Vision（ECCV），Sep 8-14，2018：87-102.
[36] LI Y，BU R，SUN M，et al.PointCNN：Convolution on x-transformed points[C]//Advances in Neural Information Processing Systems，Dec 2-8，2018：820-830.