Disparity Estimation Network with Confidence-Assisted Feature Enhancement

doi:10.3778/j.issn.1002-8331.2012-0330

Abstract

Abstract: In the disparity estimation, interference factors such as edge blur, occlusion, low light, and low texture will produce local hard areas, resulting in a reduction in effective matching and the decrease of precision of estimation results. To solve the problem, a disparity estimation network with confidence-assisted feature enhancement is proposed. In the basic disparity estimation network, a sub-network is introduced to calculate the confidence of the intermediate disparity in the estimation process, which is used for getting low error regions to enhance the feature by multiplying the corresponding feature value. Better disparity maps can be obtained from enhanced cost volume. The network is trained on the Scene Flow dataset and compared with some methods in recent years. The experimental result shows that the proposed method works better than recent classical methods. Compared with baseline method, the end-point-error decreases by 0.032?9 pixel and the weight of pixels whose errors are larger than 1 pixel decreases by 0.4 percentage points, which proves the effectiveness of the proposed method.

Key words: stereo matching, deep learning, disparity confidence, feature enhancement

摘要： 视差估计中由于边缘模糊、遮挡、低光照以及少纹理等干扰因素会产出局部困难估计区域，导致有效匹配减少，估计结果的精度降低。为此，提出了一种置信度辅助特征增强的视差估计网络，在基础的视差估计网络中引入置信度估计子网络，为网络估计过程中间的视差结果计算置信度，利用置信度图可知误差较低的点，增强代价空间中对应位置特征值的幅度，达到辅助特征增强的目的。增强后代价空间可恢复得到更精准的视差。在Scene Flow数据集上训练网络，并和近年来的一些方法进行比较。实验结果表明，该方法优于近年来的经典方法。相比于基础网络，平均误差降低了0.032?9像素；误差大于1像素的比率降低了0.4个百分点。由此可见该方法可以有效提升视差估计网络的性能。

关键词: 立体匹配, 深度学习, 视差置信度, 特征增强

LIU Boqian, LIU Jiamin, CHEN Shenglun, WANG Zhihui, LI Haojie. Disparity Estimation Network with Confidence-Assisted Feature Enhancement[J]. Computer Engineering and Applications, 2022, 58(15): 246-252.

柳博谦, 刘嘉敏, 陈圣伦, 王智慧, 李豪杰. 置信度辅助特征增强的视差估计网络[J]. 计算机工程与应用, 2022, 58(15): 246-252.

References

[1] 徐萌，陈宗海.基于视觉SLAM优化双目深度估计[C]//第21届中国系统仿真技术及其应用学术年会，2020：357-360
XU M，CHEN Z H.Optimizing binocular depth estimation based on visual SLAM[C]//21st Annual Conference of Chinese System Simulation Technology and Applications，2020：357-360
[2] SCHARSTEIN D，SZELISKI R，ZABIH R.A taxonomy and evaluation of dense two-frame stereo correspondence algorithm[C]//Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision，2001（SMBV 2001），2001.
[3] 曹之乐，严中红，王洪.双目立体视觉匹配技术综述[J].重庆理工大学学报（自然科学），2015，29（2）：70-75.
CAO Z L，YAN Z H，WANG H.Summary of binocular stereo vision matching technology[J].Journal of Chongqing University of Technology（Natural Science），2015，29（2）：70-75.
[4] VEKSLER O.Efficient graph-based energy minimization methods in computer vision[D].USA：Cornell University，1999.
[5] BOYKOV Y，VEKSLER O，ZABIH R.Fast approximate energy minimization via graph cuts[C]//International Conference on Computer Vision（ICCV），1999：377-384.
[6] 白明，庄严，王伟.双目立体匹配算法的研究与进展[J].控制与决策，2008，23（7）：721-729.
BAI M，ZHUANG Y，WANG W.Progress in binocular stereo matching algorithms[J].Control and Decision，2008，23（7）：721-729.
[7] HIRSCHMULLER H.Stereo processing by semiglobal matching and mutual information[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2008，30（2）：328-341.
[8] ?BONTAR J，LECUN Y.Stereo matching by training a convolutional neural network to compare image patches[J].Journal of Machine Learning Research，2016，17（1）：2287-2318.
[9] SEKI A，POLLEFEYS M.SGM-Nets：semi-global matching with neural networks[C]//Proceedings-30th IEEE Conference on Computer Vision and Pattern Recognition（CVPR 2017），Honolulu，HI，2017：6640-6649.
[10] MAYER N.A large dataset to train convolutional networks for disparity，optical flow，and scene flow estimation[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2016：4040-4048.
[11] DOSOVITSKIY A.FlowNet：learning optical flow with convolutional networks[C]//2015 IEEE International Conference on Computer Vision（ICCV），Santiago，2015：2758-2766.
[12] PANG J，SUN W，REN J S J，et al.Cascade residual learning：a two-stage convolutional neural network for stereo matching[C]//Proceedings-2017 IEEE International Conference on Computer Vision Workshops（ICCVW 2017），2017：878-886.
[13] YANG G，ZHAO H，SHI J，et al.SegStereo：exploiting semantic information for disparity estimation[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018.
[14] KENDALL A，MARTIROSYAN H，DASGUPTA S，et al.End-to-end learning of geometry and context for deep stereo regression[C]//IEEE International Conference on Computer Vision，2017：66-75.
[15] CHANG J R，CHEN Y S.Pyramid stereo matching network[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2018：5410-5418.
[16] GUO X，YANG K，YANG W，et al.Group-wise correlation stereo network[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：3268-3277.
[17] POGGI M，PALLOTTI D，TOSI F，et al.Guided stereo matching[C]//IEEE Conference on Computer Vision and Pattern Recognition，2019.
[18] NEWELL A，YANG K，DENG J.Stacked hourglass networks for human pose estimation[C]//European Conference on Computer Vision.Cham：Springer，2016.
[19] EGNAL G，MINTZ M，WILDES R P.A stereo confidence metric using single view imagery with comparison to five lternative approaches[J].Image and Vision Computing，2004，22（12）：943-957.
[20] EGNAL G，WILDES R P.Detecting binocular half-occlusions：empirical comparisons of five approaches[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2002，24（8）：1127-1133.
[21] KIM S，MIN D.Unified confidence estimation networks for robust stereo matching[J].IEEE Transations on Image Processing，2019，28（3）：1299-1313.
[22] YU L，WANG Y，WU Y，et al.Deep stereo matching with explicit cost aggregation sub-architecture[C]//32nd AAAI Conference on Artificial Intelligence（AAAI 2018），2018.
[23] LIANG Z.Learning for disparity estimation through feature constancy[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2018：2811-2820.