Binocular Depth Measurement Method for Desktop Interaction Scene

doi:10.3778/j.issn.1002-8331.2212-0373

Abstract

Abstract: Virtual reality interaction methods based on vision have no specific solution in desktop writing application scene. In order to provide accurate recognition of fine interactive action, a high precision three-dimensional recognition technology based on the combination of hand and pen is needed. Additionally, the depth accuracy is an important factor to the accuracy of three-dimensional recognition. Therefore, a high-precision depth measurement method in this study is provided to use in this paper. The core concept of this method is using high-resolution and close-range image pairs as input for writing interaction, and proposing the idea of cross-fusion of global and local important information to improve speed and accuracy, and reduce computing cost. In the algorithm, the region detection module is used to extract the key areas of the hand and pen tip in the image pair, and then the input is scaled according to the degree of importance. The regional feature pyramid structure is introduced to extract multi-scale semantic information. Meanwhile, disparity cascade module is used to narrow the matching range to improve the real-time performance. Finally, the experiments results confirm that this depth measurement method has high accuracy and good real-time performance in the interactive area between hand and pen tip, and can effectively assist to improve the three-dimensional recognition accuracy in further to provide better writing interactive experience. In summary, this study may provide new understandings and theoretic basis for future prospect of the depth measurement application in writing interaction.

Key words: binocular vision, deep learning, stereo matching, depth measurement, desktop interaction

摘要： 基于视觉的虚拟现实交互方式在桌面书写应用场景中尚未有针对性的解决方案，书写交互中精细动作准确识别的实现，需要一种全新的高精度手笔联合三维识别技术，其中深度准确度是影响三维识别精度的重要因素。为此提出一种高精度双目深度测量方法，该方法针对书写交互采用了高分辨率、近距离的图像对作为输入，并在算法上提出全局与局部重要信息交叉融合的思想以提升速度与精度，减少计算成本。算法使用区域检测模块提取图像对中的手部和笔尖关键区域以重要程度分尺度输入；并且引入区域特征金字塔结构结合多尺度语义信息；同时利用视差级联模块缩小匹配范围，提高网络实时性。实验证明，提出的深度测量方法在手部和笔尖交互区域精度高，实时性好，能够有效辅助提高手笔联合三维识别精度以提供更好的虚拟书写交互体验，具有广泛的应用前景。

关键词: 双目视觉, 深度学习, 立体匹配, 深度测量, 桌面交互

YE Bin, ZHU Xingshuai, YAO Kang, DING Shangshang, FU Weiwei. Binocular Depth Measurement Method for Desktop Interaction Scene[J]. Computer Engineering and Applications, 2024, 60(9): 283-291.

叶彬, 朱兴帅, 姚康, 丁上上, 付威威. 面向桌面交互场景的双目深度测量方法[J]. 计算机工程与应用, 2024, 60(9): 283-291.

References

[1] HAN S, LIU B, CABEZAS R, et al. MEgATrack: monochrome egocentric articulated hand-tracking for virtual reality[J]. ACM Transactions on Graphics (ToG), 2020, 39(4): 1-13.
[2] 鲁光男. 基于交互式视景的虚拟现实单目深度信息提取[J]. 计算机仿真, 2020, 37(12): 382-385.
LU G N. Virtual reality monocular depth information extraction based on interactive view[J]. Computer Simulation, 2020, 37(12): 382-385.
[3] MOON G, YU S I, WEN H, et al. Interhand2. 6m: a dataset and baseline for 3D interacting hand pose estimation from a single rgb image[C]//European Conference on Computer Vision. Cham: Springer, 2020: 548-564.
[4] 张方义. 基于双目视觉的智能车辆障碍物检测系统研究[D]. 青岛: 山东科技大学, 2019.
ZHANG F Y. Research on intelligent vehicle obstacle detection system based on binocular vision[D]. Qingdao: Shandong University of Science and Technology, 2019.
[5] 贾鑫. 基于深度学习的双目三维物体稀疏与稠密点云重建[D]. 天津: 天津理工大学, 2022.
JIA X. Binocular 3D object sparse and dense point cloud reconstruction based on deep learning[D]. Tianjin: Tianjin University of Technology, 2022.
[6] SCHARSTEIN D, SZELISKI R. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms[C]//Proceedings IEEE Workshop on Stereo and Multi-Baseline Vision (SMBV 2001), 2001.
[7] 尹晨阳, 职恒辉, 李慧斌. 基于深度学习的双目立体匹配方法综述[J]. 计算机工程, 2022, 48(10): 1-12.
YIN C Y, ZHI H H, LI H B. Survey of binocular stereo-matching methods based on deep learning[J]. Computer Engineering, 2022, 48(10): 1-12.
[8] ?BONTAR J, LECUN Y. Stereo matching by training a convolutional neural network to compare image patches[J]. arXiv:1510.05970, 2015.
[9] SONG X, ZHAO X, HU H, et al. Edgestereo: a context integrated residual pyramid network for stereo matching[C]//Asian Conference on Computer Vision. Cham: Springer, 2018: 20-35.
[10] CAO Y, XU J, LIN S, et al. GCNet: non-local networks meet squeeze-excitation networks and beyond[C]//2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), 2020.
[11] CHANG J R, CHEN Y S. Pyramid stereo matching network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 5410-5418.
[12] GUO X, YANG K, YANG W, et al. Group-wise correlation stereo network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 3273-3282.
[13] DUGGAL S, WANG S, MA W C, et al. DeepPruner: learning efficient stereo matching via differentiable patch match[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 4384-4393.
[14] ZHANG F, PRISACARIU V, YANG R, et al. GA-Net: guided aggregation net for end-to-end stereo matching[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 185-194.
[15] XU H, ZHANG J. AANet: adaptive aggregation network for efficient stereo matching[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 1959-1968.
[16] HE K, ZHANG X, REN S, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916.
[17] CHEN L C, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation[J]. arXiv:1706.05587, 2017.
[18] LIN T Y, DOLLáR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 2117-2125.
[19] GU X, FAN Z, ZHU S, et al. Cascade cost volume for high-resolution multi-view stereo and stereo matching[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 2495-2504.
[20] ZHANG K, ZHANG Z, LI Z, et al. Joint face detection and alignment using multitask cascaded convolutional networks[J]. IEEE Signal Processing Letters, 2016, 23(10): 1499-1503.
[21] DU H, LI Y, SUN Y, et al. SRH-Net: stacked recurrent hourglass network for stereo matching[J]. IEEE Robotics and Automation Letters, 2021, 6(4): 8005-8012.
[22] SHAMSAFAR F, WOERZ S, RAHIM R, et al. Mobilestereonet: towards lightweight deep networks for stereo matching[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022: 2417-2426.
[23] XU G, CHENG J, GUO P, et al. Attention concatenation volume for accurate and efficient stereo matching[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 12981-12990.