Review of 2D-3D Fusion Deep Completion of Deep Learning

doi:10.3778/j.issn.1002-8331.2209-0284

Abstract

Abstract: The purpose of depth map completion is to predict dense pixel-level depth from sparse maps captured by depth sensors. It plays a vital role in a variety of applications such as autonomous driving, 3D reconstruction, augmented reality, and robot navigation. Recent success in this task proves that deep learning-based 2D-3D fusion depth map completion technology has become a mainstream scheme in this field. This paper discusses the research status of this method in the industry in recent years, analyzes the data sets and evaluation indicators commonly used in the completion task, and the processing methods of noise and sparse data obtained by sensors. The fusion methods of the two modal appearance features are divided into：early fusion, late fusion and multi-level fusion, and the characteristics and problems are compared from the perspective of extracting geometric clues and multi-task learning. The development prospect and possible research directions of depth map completion are prospected.

Key words: deep learning, depth map completion, autonomous driving, 3D reconstruction, 2D-3D fusion

摘要： 深度图补全的目的是从深度传感器捕获的稀疏图预测密集像素级深度。它在自动驾驶、三维重建、增强现实和机器人导航等各种应用中发挥着至关重要的作用。最近在这项任务上的成功证明基于深度学习的2D-3D融合深度图补全技术成为该领域的主流方案。论述了该方法近年在业界的研究现状，分析了补全任务常用的数据集与评价指标以及对传感器获取的噪声和稀疏数据的处理方法。将两个模态外观特征的融合方式分为：早期融合、后期融合和多级融合，从提取几何线索和多任务学习角度出发进行归纳分析并对其优势和局限性进行对比。对深度图补全的发展前景和可能的研究方向进行了展望。

关键词: 深度学习, 深度图补全, 自动驾驶, 三维重建, 2D-3D融合

BAI Yu, LIANG Xiaoyu, AN Shengbiao. Review of 2D-3D Fusion Deep Completion of Deep Learning[J]. Computer Engineering and Applications, 2023, 59(13): 17-32.

白宇, 梁晓玉, 安胜彪. 深度学习的2D-3D融合深度补全综述[J]. 计算机工程与应用, 2023, 59(13): 17-32.

References

[1] ZHANG Y，FUNKHOUSER T.Deep depth completion of a single RGB-D image[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：175-185.
[2] LAINA I，RUPPRECHT C，BELAGIANNIS V，et al.Deeper depth prediction with fully convolutional residual networks[C]//Proceedings of the 4th International Conference on 3D Vision（3DV），2016：239-248.
[3] EIGEN D，PUHRSCH C，FERGUS R.Depth map prediction from a single image using a multi-scale deep network[C]//Advances in Neural Information Processing Systems，2014.
[4] ZHAO C，SUN Q，ZHANG C，et al.Monocular depth estimation based on deep learning：an overview[J].Science China Technological Sciences，2020，63（9）：1612-1627.
[5] GEIGER A，LENZ P，URTASUN R.Are we ready for autonomous driving? the kitti vision benchmark suite[C]//Proceedings of the IEEE Conference on Computer Vision and pattern Recognition，2012：3354-3361.
[6] KU J，HARAKEH A，WASLANDER S L.In defense of classical image processing：fast depth completion on the cpu[C]//Proceedings of the 15th Conference on Computer and Robot Vision（CRV），2018：16-22.
[7] ZHAO S，GONG M，FU H，et al.Adaptive context-aware multi-modal network for depth completion[J].IEEE Transactions on Image Processing，2021，30：5264-5276.
[8] HUANG Z，FAN J，CHENG S，et al.HMS-Net：hierarchical multi-scale sparsity-invariant network for sparse depth completion[J].IEEE Transactions on Image Processing，2019，29：3429-3441.
[9] VAN GANSBEKE W，NEVEN D，DE BRABANDERE B，et al.Sparse and noisy lidar completion with RGB guidance and uncertainty[C]//Proceedings of the 16th International Conference on Machine Vision Applications（MVA），2019：1-6.
[10] CHEN Y，YANG B，LIANG M，et al.Learning joint 2D-3D representations for depth completion[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：10023-10032.
[11] XIONG X，XIONG H，XIAN K，et al.Sparse-to-dense depth completion revisited：sampling strategy and graph construction[C]//Proceedings of the European Conference on Computer Vision，2020：682-699.
[12] LLI A，YUAN Z，LING Y，et al.A multi-scale guided cascade hourglass network for depth completion[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision，2020：32-40.
[13] HU M，WANG S，LI B，et al.PENet：towards precise and efficient image guided depth completion[C]//Proceedings of the IEEE International Conference on Robotics and Automation（ICRA），2021：13656-13662.
[14] ZHU Y，DONG W，LI L，et al.Robust depth completion with uncertainty-driven loss functions[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2022：3626-3634.
[15] TANG J，TIAN F P，FENG W，et al.Learning guided convolutional network for depth completion[J].IEEE Transactions on Image Processing，2020，30：1116-1129.
[16] LEE S，LEE J，KIM D，et al.Deep architecture with cross guidance between single image and sparse lidar data for depth completion[J].IEEE Access，2020，8：79801-79810.
[17] LIU L，SONG X，LYU X，et al.FCFR-Net：feature fusion based coarse-to-fine residual learning for depth completion[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2021：2136-2144.
[18] YAN Z，WANG K，LI X，et al.RigNet：repetitive image guided network for depth completion[J].arXiv：2107. 13802，2021.
[19] ELDESOKEY A，FELSBERG M，KHAN F S.Confidence propagation through cnns for guided sparse depth regression[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2019，42（10）：2423-2436.
[20] UHRIG J，SCHNEIDER N，SCHNEIDER L，et al.Sparsity invariant CNNs[C]//Proceedings of the International Conference on 3D Vision（3DV），2017：11-20.
[21] QIU J，CUI Z，ZHANG Y，et al.Deeplidar：deep surface normal guided depth prediction for outdoor scene from sparse lidar data and single color image[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：3313-3322.
[22] MA F，CAVALHEIRO G V，KARAMAN S.Self-supervised sparse-to-dense：self-supervised depth completion from lidar and monocular camera[C]//Proceedings of the International Conference on Robotics and Automation（ICRA），2019：3288-3295.
[23] MA F，KARAMAN S.Sparse-to-dense：depth prediction from sparse depth samples and a single image[C]//Proceedings of the IEEE International Conference on Robotics And Automation（ICRA），2018：4796-4803.
[24] PARK J，JOO K，HU Z，et al.Non-local spatial propagation network for depth completion[C]//Proceedings of the European Conference on Computer Vision，2020：120-136.
[25] LU K，BARNES N，ANWAR S，et al.Depth completion auto-encoder[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision Workshops（WACVW），2022：63-73.
[26] CHENG X，WANG P，YANG R.Learning depth with convolutional spatial propagation network[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2019，42（10）：2361-2379.
[27] ZOU N，XIANG Z，CHEN Y，et al.Simultaneous semantic segmentation and depth completion with constraint of boundary[J].Sensors，2020，20（3）：635.
[28] 厉佳男.基于置信度传播、语义信息和特征增强的深度补全研究[D].杭州：浙江大学，2020.
LI J N.Deep completion based on confidence propagation，semantic information and feature enhancement[D].Hangzhou：Zhejiang University，2020.
[29] ZHANG C，TANG Y，ZHAO C，et al.Multitask gans for semantic segmentation and depth completion with cycle consistency[J].IEEE Transactions on Neural Networks and Learning Systems，2021，32（12）：5404-5415.
[30] WONG A，FEI X，TSUEI S，et al.Unsupervised depth completion from visual inertial odometry[J].IEEE Robotics and Automation Letters，2020，5（2）：1899-1906.
[31] LOPEZ-RODRIGUEZ A，BUSAM B，MIKOLAJCZYK K.Project to adapt：domain adaptation for depth completion from noisy and sparse sensor data[C]//Proceedings of the Asian Conference on Computer Vision，2020.
[32] DONG X，GARRATT M A，ANAVATTI S G，et al.Towards real-time monocular depth estimation for robotics：a survey[J].arXiv：2111.08600，2021.
[33] YANG Y，WONG A，SOATTO S.Dense depth posterior（DDP） from single image and sparse range[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：3353-3362.
[34] LAGA H，JOSPIN L V，BOUSSAID F，et al.A survey on deep learning techniques for stereo-based depth estimation[J].arXiv：2006.02535，2020.
[35] CHEN H，YANG H，ZHANG Y.Depth completion using geometry-aware embedding[C]//Proceedings of the International Conference on Robotics and Automation（ICRA），2022：8680-8686.
[36] ZHANG Y，WEI P，LI H，et al.Multiscale adaptation fusion networks for depth completion[C]//Proceedings of the International Joint Conference on Neural Networks（IJCNN），2020：1-7.
[37] JARITZ M，DE CHARETTE R，WIRBEL E，et al.Sparse and dense data with CNNs：depth completion and semantic segmentation[C]//Proceedings of the International Conference on 3D Vision（3DV），2018：52-60.
[38] ZOPH B，VASUDEVAN V，SHLENS J，et al.Learning transferable architectures for scalable image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：8697-8710.
[39] BHOI A.Monocular depth estimation：a survey[J].arXiv：1901.09402，2019.
[40] SILBERMAN N，HOIEM D，KOHLI P，et al.Indoor segmentation and support inference from RGB-D images[C]//Proceedings of the European Conference on Computer Vision.Berlin，Heidelberg：Springer，2012：746-760.
[41] CHANG A，DAI A，FUNKHOUSER T，et al.Matterport3D：learning from RGB-D data in indoor environments[J].arXiv：1709.06158，2017.
[42] YU Q，CHU L，WU Q，et al.Grayscale and normal guided depth completion with a low-cost lidar[C]//Proceedings of the 2021 IEEE International Conference on Image Processing（ICIP），2021：979-983.