Multi-Level 3D Point Cloud Completion with Dual-Branch Structure

doi:10.3778/j.issn.1002-8331.2212-0355

Abstract

Abstract: In order to alleviate the problem that the existing point cloud completion methods are difficult to balance local features and global features in the feature extraction process, this paper proposes a multi-level point cloud completion algorithm with double branch structure. Two independent branch networks are used to extract the local feature information and global feature information of the input point cloud respectively, and the two feature information are concatenated to form a feature vector. The five levels combinate perceptron is used to map the feature vector into multiple dimensions, and the multi-dimensional feature information is extracted and integrated into the final feature vector. Then, the pyramid structure is used to decode the final feature vector in 256, 512 and 1 024 feature dimensions, and the point clouds with three different resolutions are predicted. Finally, the discriminator network is introduced to optimize the network by jointly training the adversarial loss generated by the discriminator and the completion loss generated by the hierarchical reconstruction point cloud. Experiments on ShapeNet dataset show that the algorithm significantly improves the accuracy of point cloud completion. In addition, a relatively complete object shape can be recovered when a large area of point cloud is missing.

Key words: three-dimensional point cloud, shape completion, deep learning, dual-branch structure, discriminator network

摘要： 为了缓解现有点云补全方法在特征提取过程中很难平衡局部特征和全局特征的问题，提出了一种双分支结构的多层级点云补全算法。利用两个独立的分支网络分别提取出输入点云的局部特征信息和全局特征信息，再将两种特征信息进行拼接形成特征向量。使用五层联合感知机将特征向量映射成多个维度，进而提取多维特征信息并将其整合成最终特征向量。采用金字塔结构在256、512、1?024特征维度上对最终特征向量进行特征解码，预测三种不同分辨率的点云。引入鉴别器网络，通过联合训练鉴别器产生的对抗损失和分层重建点云产生的补全损失去优化网络。在ShapeNet数据集上进行实验，算法显著提升了点云补全精度，并且在缺失大面积点云时也能恢复出较为完善的物体形状。

关键词: 三维点云, 形状补全, 深度学习, 双分支结构, 鉴别器网络

QIU Yunfei, WANG Yifan. Multi-Level 3D Point Cloud Completion with Dual-Branch Structure[J]. Computer Engineering and Applications, 2024, 60(9): 272-282.

邱云飞, 王宜帆. 双分支结构的多层级三维点云补全[J]. 计算机工程与应用, 2024, 60(9): 272-282.

References

[1] 沙浩, 刘越, 王涌天, 等. 基于二维图像和三维几何约束神经网络的单目室内深度估计方法[J]. 光学学报, 2022, 42(19): 39-49.
SHA H, LIU Y, WANG Y T, et al. Monocular indoor depth estimation method based on neural networks with constraints on two-dimensional images and three-dimensional geometry[J]. Acta Optica Sinica, 2022, 42(19): 39-49.
[2] 胡春生, 闫小鹏, 魏红星, 等. 基于立体视觉的目标检测与轨迹预测研究综述[J]. 计算机工程与应用, 2022, 58(3): 50-65.
HU C S, YAN X P, WEI H X, et al. Survey of target detection and trajectory prediction based on stereo vision[J]. Computer Engineering and Applications, 2022, 58(3): 50-65.
[3] 马建红, 王稀瑶, 陈永霞, 等. 自动驾驶中图像与点云融合方法研究综述[J]. 郑州大学学报(理学版), 2022, 54(6): 24-33.
MA J H, WANG X Y, CHEN Y X, et al. A review of research on image and point cloud fusion methods in automatic driving[J]. Journal of Zhengzhou University (Science Edition), 2022, 54(6): 24-33.
[4] 张王, 高晓蓉, 李金龙, 等. 基于二步相移法的钢轨三维面形复原[J]. 激光与光电子学进展, 2022, 59(10): 1015003.
ZHANG W, GAO X R, LI J L, et al. Three-dimensional surface reconstruction of rail based on two-step phase-shift method[J]. Laser & Optoelectronics Progress, 2022, 59(10): 1015003.
[5] 丁子轩, 张娟, 李想, 等. 基于注意力引导的轻量级图像超分辨率网络[J]. 激光与光电子学进展, 2023, 60(14): 95-103.
DING Z X, ZHANG J, LI X, et al. Lightweight attention-guided network for image super-resolution[J]. Laser & Optoelectronics Progress, 2023, 60(14): 95-103.
[6] CHARLES R Q, HAO S, MO K C, et al. PointNet: deep learning on point sets for 3D classification and segmentation[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 21-26, 2017, Honolulu, HI, USA. New York: IEEE, 2017: 77-85.
[7] CHARLESR Q, LI Y, HAO S, et al. PointNet++: deep hierarchical feature learning on point sets in a metric space[C]//2008 IEEE International Conference on Neural Information Processing Systems, Long Beach, CA, USA, 2017: 5105-5114.
[8] YUAN W T, KHOT T, HELD D, et al. PCN: point completion network[C]//2018 International Conference on 3D Vision (3DV), September 5-8, 2018, Verona, Italy. New York: IEEE, 2018: 728-737.
[9] HUANG Z, YU Y, XU J, et al. PF-Net: point fractal network for 3D point cloud completion[C]//2020 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 2020: 7662-7670.
[10] LIN T Y, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017: 2117-2125.
[11] ZHANG J, CHEN X, CAI Z, et al. Unsupervised 3D shape completion through GAN inversion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 1768-1777.
[12] 孙娅楠, 林文斌. 梯度下降法在机器学习中的应用[J]. 苏州科技大学学报 (自然科学版), 2018, 35(2): 26-31.
SUN Y N, LIN W B. Application of gradient descent method in machine learning[J]. Journal of Suzhou University of Science and Technology (Natural Science Edition), 2018, 35(2): 26-31.
[13] WEN X, XIANG P, HAN Z, et al. PMP-Net: point cloud completion by learning multi-step point moving paths[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 7443-7452.
[14] WEN X, XIANG P, HAN Z, et al. PMP-Net++: point cloud completion by transformer-enhanced multi-step point moving paths[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(1): 852-867.
[15] TANG J, GONG Z, YI R, et al. LAKe-Net: topology-aware point cloud completion by localizing aligned keypoints[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 1726-1735.
[16] TCHAPMI L I, KOSARAJU V, REZATOFIFIGHI H, et al. TopNet: structural point cloud decoder[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 383-392.
[17] YU L, LI X, FU C W, et al. PU-Net: point cloud upsampling network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 2790-2799.
[18] GROUEIX T, FISHER M, KIM V G, et al. A papier-maché approach to learning 3D surface generation[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018: 216-224.
[19] WANG Y, SUN Y B, LIU Z W, et al. Dynamic graph CNN for learning on point clouds[J]. ACM Transactions on Graphics, 2018, 38(5): 1-12.
[20] HANG W, MIAO Y, FU R C. Point cloud completion using multiscale feature fusion and cross-regional attention[J]. Image and Vision Computing, 2021, 111: 104193.
[21] HUA B S, TRAN M K, YEUNG S K. Pointwise convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 984-993.
[22] CHIBANE J, ALLDIECK T, PONS-MOLL G. Implicit functions in feature space for 3D shape reconstruction and completion[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, 2020: 6970-6981.
[23] XIE H Z, YAO H X, ZHOU S C, et al. GRNet: gridding residual network for dense point cloud completion[C]//Proceedings of the 16th European Conference on Computer Vision. Glasgow, Scotland: Springer, 2020: 365-381.
[24] WANG X, ANG M H, LEE G H. Voxel-based network for shape completion by leveraging edge generation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 13189-13198.
[25] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Long Beach, USA: NIPS, 2017: 6000-6010.
[26] PAN X R, XIA Z F, SONG S J, et al. 3D object detection with pointformer[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 2021: 7463-7472.
[27] ZHAO H S, JIANG L, JIA J Y, et al. Point transformer[EB/OL]. [2021-06-20]. https: //arxiv. org/pdf/2012. 09164. pdf.
[28] YU X M, RAO Y M, WANG Z Y, et al. Pointr: diverse point cloud completion with geometry-aware transformers[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, Montreal, QC, Canada, 2021: 12478-12487.
[29] LIN J J, RICKERT M, PERZYLO A, et al. Pctmanet: point cloud transformer with morphing atlas-based point generation network for dense point cloud completion[C]//Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Prague, Czech Republic, 2021: 5657-5663.
[30] 麻卫峰, 王金亮, 张建鹏, 等. 一种改进法向量估算的点云特征提取[J]. 测绘科学, 2021, 46(11): 84-90.
MA W F, WANG L J, ZHANG J P, et al. Feature extraction from point cloud based on improved normal vector[J]. Science of Surveying and Mapping, 2021, 46(11): 84-90.
[31] ACHLIOPTAS P, DIAMANTI O, MITLIAGKAS I, et al. Learning representations and generative models for 3d point clouds[C]//International Conference on Machine Learning, 2018: 40-49.
[32] 曾安, 彭杰威, 刘畅, 等. 基于多尺度几何感知 Transformer 的植物点云补全网络[J]. 农业工程学报, 2022, 38(4): 198-205.
ZENG A, PENG J W, LIU C, et al. Plant point cloud completion network based on multi-scale geometry-aware point Transformer[J]. Transactions of the Chinese Society of Agricultural Engineering, 2022, 38(4): 198-205.
[33] 刘鹏南, 徐冬冬, 白春梦. 基于尺度不变特征变换的异源图像配准方法[J]. 激光与光电子学进展, 2021, 58(24): 164-173.
LIU P N, XU D D, BAI C M. Scale-invariant feature transform-based heterogeneous image registration method[J]. Laser & Optoelectronics Progress, 2021, 58(24): 164-173.
[34] FAN H, HAO S, GUIBAS L. A point set generation network for 3D object reconstruction from a single image[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[35] CHANG A X, FUNKHOURSER T, GUIBAS L, et al. Shape-
Net: an information-rich 3D model repository[J]. arXiv:1512.03012, 2015.
[36] ZHOU J, WANG H, WEI J, et al. Adaptive moment estimation for polynomial nonlinear equalizer in PAM8-based optical interconnects[J]. Optics Express, 2019, 27(22): 32210-32216.
[37] WU S, LI G, DENG L, et al. L1-norm batch normalization for efficient training of deep neural networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2018, 30(7): 2043-2051.
[38] ZENG Z, HUANG T, ZHENG W X. Multistability of recurrent neural networks with time-varying delays and the piecewise linear activation function[J]. IEEE Transactions on Neural Networks, 2010, 21(8): 1371-1377.
[39] ZHAO Y, BIRDAL T, DENG H, et al. 3D point capsule networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 1009-1018.
[40] XIANG P, WEN X, LIU Y S, et al. SnowflakeNet: point cloud completion by snowflake point deconvolution with skip-transformer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 5499-5509.