Computer Engineering and Applications ›› 2023, Vol. 59 ›› Issue (13): 17-32.DOI: 10.3778/j.issn.1002-8331.2209-0284

• Research Hotspots and Reviews • Previous Articles     Next Articles

Review of 2D-3D Fusion Deep Completion of Deep Learning

BAI Yu, LIANG Xiaoyu, AN Shengbiao   

  1. School of Information Science and Engineering, Hebei University of Science and Technology, Shijiazhuang 050018, China
  • Online:2023-07-01 Published:2023-07-01

深度学习的2D-3D融合深度补全综述

白宇,梁晓玉,安胜彪   

  1. 河北科技大学 信息科学与工程学院,石家庄 050018

Abstract: The purpose of depth map completion is to predict dense pixel-level depth from sparse maps captured by depth sensors. It plays a vital role in a variety of applications such as autonomous driving, 3D reconstruction, augmented reality, and robot navigation. Recent success in this task proves that deep learning-based 2D-3D fusion depth map completion technology has become a mainstream scheme in this field. This paper discusses the research status of this method in the industry in recent years, analyzes the data sets and evaluation indicators commonly used in the completion task, and the processing methods of noise and sparse data obtained by sensors. The fusion methods of the two modal appearance features are divided into:early fusion, late fusion and multi-level fusion, and the characteristics and problems are compared from the perspective of extracting geometric clues and multi-task learning. The development prospect and possible research directions of depth map completion are prospected.

Key words: deep learning, depth map completion, autonomous driving, 3D reconstruction, 2D-3D fusion

摘要: 深度图补全的目的是从深度传感器捕获的稀疏图预测密集像素级深度。它在自动驾驶、三维重建、增强现实和机器人导航等各种应用中发挥着至关重要的作用。最近在这项任务上的成功证明基于深度学习的2D-3D融合深度图补全技术成为该领域的主流方案。论述了该方法近年在业界的研究现状,分析了补全任务常用的数据集与评价指标以及对传感器获取的噪声和稀疏数据的处理方法。将两个模态外观特征的融合方式分为:早期融合、后期融合和多级融合,从提取几何线索和多任务学习角度出发进行归纳分析并对其优势和局限性进行对比。对深度图补全的发展前景和可能的研究方向进行了展望。

关键词: 深度学习, 深度图补全, 自动驾驶, 三维重建, 2D-3D融合