Task-Driven Point Cloud Downsampling Method Based on Light-Transformer

doi:10.3778/j.issn.1002-8331.2303-0188

Abstract

Abstract: Point cloud is one of the most important 3D data format that objectively represents the real world. However, the huge amount of data of point cloud limits its wider application. In order to simplify the point cloud and improve the efficiency of downstream applications, a task-driven point cloud downsampling method based on lightweight Transformer is proposed. The network includes feature extraction module and soft sampling module. In the feature extraction module, the state-of-the-art Transformer model is used to learn point cloud features, which is designed as a lightweight structure considering computing and storage resources. In the soft sampling module, MLP and Gumbel-Softmax are used to simulate the actual sampling process and obtain the downsampled point cloud. To make the downsampled point cloud suitable for subsequent application tasks, a joint loss function including task loss, sampling loss, and constraint loss is constructed for end-to-end training. To simplify training and facilitate practical application, on the basis of a task-driven point cloud downsampling method, a multi-rate downsampling method is also proposed, which uses a progressive structure and combines multiple sets of sampling losses to obtain point clouds at multiple sampling rates with a single model. Experiments of point cloud classification task on ModelNet40 dataset and reconstruction task on ShapeNetCore55 dataset show that the proposed method not only simplify the point cloud but also ensuring good task performance, especially when the number of downsampled points is small, the task performance is higher than that of similar methods.

Key words: 3D point cloud, downsampling, task-driven, multi-rate, Transformer

摘要： 点云是一种重要的三维数据格式，能直观地描绘真实世界，然而点云的巨大数据量限制了其更加广泛的应用。为了简化点云并提高下游应用效率，提出了一种基于轻量级Transformer的任务驱动点云下采样方法。该网络包括特征提取模块和软采样模块，在特征提取模块中采用最先进的Transformer模型学习点云特征，考虑到计算和存储资源有限，将其设计为轻量化结构；在软采样模块中利用MLP和Gumbel-Softmax来模拟实际采样过程，得到下采样点云。为使采样点云适合后续应用任务，构造了一个包含任务损失、采样损失和约束损失的联合损失函数用于网络端到端训练。此外，为简化训练并方便实际应用，在基于轻量化Transformer的任务驱动点云下采样网络的基础上，还提出了多倍率下采样方法，它采用渐进式结构，结合多组采样损失，实现一个模型得到多个采样率下的点云。通过在ModelNet40和ShapeNetCore55数据集上进行点云分类任务和重建任务实验表明，所提方法在简化点云数量的同时，分类精度和重建精度得到良好保证，尤其是下采样点数较少时，相比于同类算法，任务性能更高。

关键词: 三维点云, 下采样, 任务驱动, 多倍率, Transformer

YANG Yakun, WANG Anhong, FENG Zewen. Task-Driven Point Cloud Downsampling Method Based on Light-Transformer[J]. Computer Engineering and Applications, 2023, 59(21): 159-166.

杨亚坤, 王安红, 冯泽文. 任务驱动的轻量Transformer点云下采样方法[J]. 计算机工程与应用, 2023, 59(21): 159-166.

References

[1] 王文曦，李乐林.深度学习在点云分类中的研究综述[J].计算机工程与应用，2022，58（1）：26-40.
WANG W X，LI L L.Review of deep learning in point cloud classification[J].Computer Engineering and Applications，2022，58（1）：26-40.
[2] PAULY M，GROSS M，KOBBELT L P.Efficient simplification of point-sampled surfaces[C]//IEEE Visualization，Boston，Oct 27-Nov 1 2002.Piscataway：IEEE Press，2002：163-170.
[3] MOENNING C，DODGSON N A.Fast marching farthest point sampling for implicit surfaces and point clouds[J].Computer Laboratory Technical Report，2003，565：1-12.
[4] KATZ S，TAL A.Improving the visual comprehension of point sets[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Oregon，Jun 25-27 2003.Piscataway：IEEE Press，2013：121-128.
[5] YING X，XIN S Q，SUN Q，et al.An intrinsic algorithm for parallel poisson disk sampling on arbitrary surfaces[J].IEEE Transactions on Visualization and Computer Graphics，2013，19（9）：1425-1437.
[6] CHEN S，TIAN D，FENG C，et al.Fast resampling of three-dimensional point clouds via graphs[J].IEEE Transactions on Signal Processing，2017，66（3）：666-681.
[7] HINKS T，CARR H，TRUONG-HONG L，et al.Point cloud data conversion into solid models via point-based voxelization[J].Journal of Surveying Engineering，2013，139（2）：72-83.
[8] YANG J，ZHANG Q，NI B，et al.Modeling point clouds with self-attention and gumbel subset sampling[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，Long Beach，Jun 15-20 2019.Piscataway：IEEE Press，2019：3323-3332.
[9] NEZHADARYA E，TAGHAVI E，RAZANI R，et al.Adaptive hierarchical down-sampling for point cloud classification[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，Seattle，Jun 13-19 2020.Piscataway：IEEE Press，2020：12956-12964.
[10] DOVRAT O，LANG I，AVIDAN S.Learning to sample[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，Long Beach，Jun 15-20 2020.Piscataway：IEEE Press，2019：2760-2769.
[11] LANG I，MANOR A，AVIDAN S.Samplenet：differentiable point cloud sampling[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，Seattle，Jun 13-19 2020.Piscataway：IEEE Press，2020：7578-7588.
[12] QI C R，SU H，MO K，et al.Pointnet：deep learning on point sets for 3d classification and segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Honolulu，Jul 21-26 2017.Piscataway：IEEE Press，2017：652-660.
[13] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems，2017.
[14] QIAN Y，HOU J，ZHANG Q，et al.MOPS-Net：a matrix optimization-driven network fortask-oriented 3D point cloud downsampling[J].arXiv：2005.00383，2020.
[15] WANG X，JIN Y，CEN Y，et al.PST-NET：point cloud sampling via point-based transformer[C]//11th International Conference on Image and Graphics，Haikou，Aug 6-8 2021.Switzerland：Springer International Publishing，2021：57-69.
[16] ZHAO H，JIANG L，JIA J，et al.Point transformer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，Montreal，Oct 10-17 2021.Piscataway：IEEE Press，2021：16259-16268.
[17] GUO M H，CAI J X，LIU Z N，et al.PCT：point cloud transformer[J].Computational Visual Media，2021，7：187-199.
[18] WU Z，SONG S，KHOSLA A，et al.3d shapenets：a deep representation for volumetric shapes[C]]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Boston，Jun 7-12 2015.Piscataway：IEEE Press，2015：1912-1920.
[19] ACHLIOPTAS P，DIAMANTI O，MITLIAGKAS I，et al.Learning representations and generative models for 3d point clouds[C]//Proceedings of the 35th International Conference on Machine Learning，Stockholm，July 10-15 2018.New York：ACM，2018：40-49.
[20] CHANG A X，FUNKHOUSER T，GUIBAS L，et al.Shapenet：an information-rich 3d model repository[J].arXiv：1512.03012，2015.
[21] QI C R，YI L，SU H，et al.Pointnet++：deep hierarchical feature learning on point sets in a metric space[C]//Advances in Neural Information Processing Systems，2017.