Gesture Interaction and Linkage System of Flat-Panel Display and Integral Imaging for Bones

doi:10.3778/j.issn.1002-8331.2211-0367

Abstract

Abstract: Flat-panel display interacted with the mouse and keyboard has difficulty in presenting multi-view spatial information of the human bones.Moreover, it has problems of heavily taking place and bacterial infections.Integral imaging is widely claimed to be a naked-eye three-dimensional display with the highest potential of development.A linkage system of flat-panel display and integral imaging supporting gesture interaction is proposed.The real-time performance of Optix and Leap Motion, respectively in ray casting and accurate gesture recognition, is combined.The mathematical relationship between the pose of virtual camera array and the gesture data is considered.By immediate feedback of gesture information, parameter updating of virtual cameras, and efficient rendering of content shown, synchronous adjusting the viewpoint and scale of the bone’s two-dimensional view and naked-eye three-dimensional image by viewer’s gestures is allowed.The results indicate that, the framerate of this system can reach 30 frames per second on single GTX 1080Ti GPU, when the two- and three-dimensional（26 view points） images both have scree resolution of 2?160×1?280.The effective and fluent telekinesis interaction as well as mixture and linkage display help to improve of viewing experience, sensory impact, and practical worth three-dimensional bone visualization.

Key words: integral imaging, gesture interaction, double-screen linkage, three-dimensional medical imaging

摘要： 基于鼠键控制的医用平面显示不易同时呈现骨骼的多视角空间信息，还存在占用空间和细菌感染的问题；集成成像被普遍认为是发展潜力最大的裸眼三维显示技术。提出一种支持手势交互的平面显示与集成成像联动系统，挖掘Optix和Leap Motion分别在光线投射和精准手势识别方面的实时性，根据虚拟相机阵列的位姿与手势数据之间的数学关系，通过即时的手势指令检测、虚拟相机位姿更迭及显示内容快速渲染，允许观看者利用手势同步调整骨骼二维视图和裸眼三维影像的视角和尺度。实验结果表明，该系统在二维与立体（26视点）画面的屏幕分辨率皆为2?160×1?280的情况下，在单块GTX 1080Ti GPU上的帧率达到36帧/s，有效和流畅的隔空交互、混合联动显示有助于提升骨骼三维可视化的观看体验、感官冲击和实际应用价值。

关键词: 集成成像, 手势交互, 双屏联动, 三维医学成像

WEI Jian, GUO Xiaoyuan, JI Fenglei, GUO Shaoyuan, HU Xiaohan, JIA Yong, WANG Shigang. Gesture Interaction and Linkage System of Flat-Panel Display and Integral Imaging for Bones[J]. Computer Engineering and Applications, 2023, 59(19): 285-293.

韦健, 郭校源, 籍风磊, 郭绍源, 胡笑含, 贾勇, 王世刚. 骨骼平面显示与集成成像手势交互联动系统[J]. 计算机工程与应用, 2023, 59(19): 285-293.

References

[1] 唐慧，杨奎，鲍旭东.医学三维图像显示平台的开发与研究[J].计算机工程与应用，2006，42（3）：212-214.
TANG H，YANG K，BAO X D.Design and implementation of a three dimensional medical data visualization platform[J].Computer Engineering and Applications，2006，42（3）：212-214.
[2] KIM D，KYE H，LEE J，et al.Confidence-controlled local isosurfacing[J].IEEE Transactions on Visualization and Computer Graphics，2021，27（1）：29-42.
[3] 胡建青，赵悟翔，马晓莉，等.一种弱化莫尔条纹的狭缝光栅屏裸眼3D显示器[J].光学学报，2018，38（10）：194-198.
HU J Q，ZHAO W X，MA X L，et al.A naked-eye 3D LED display based on a parallax barrier with weak moire fringes[J].Acta Optica Sinica，2018，38（10）：194-198.
[4] 陈载清，陈凯，黄小乔，等.光栅式自由立体显示器的颜色特性化实验[J].光学学报，2019，39（5）：429-436.
CHEN Z Q，CHEN K，HUANG X Q，et al.Color characterization of barrier-type autostereoscopic displays[J].Acta Optica Sinica，2019，39（5）：429-436.
[5] CHEN F，CUI X，HAN B，et al.Augmented reality navigation for minimally invasive knee surgery using enhanced arthroscopy[J].Computer Methods and Programs in Biomedicine，2021，201（3）：105952.
[6] 李炜，李海峰，徐良，等.基于体感检测器件的悬浮真三维显示力触觉交互[J].光学学报，2020，40（9）：127-132.
LI W，LI H F，XU L，et al.Suspended three-dimensional display haptic interaction based on leap motion[J].Acta Optica Sinica，2020，40（9）：127-132.
[7] MA C，CHEN G，ZHANG X，et al.Moving-tolerant augmented reality surgical navigation system using autostereoscopic three-dimensional image overlay[J].IEEE Journal of Biomedical and Health Informatics，2019，23（6）：2483-2493.
[8] 苏忱，李海峰.应用于混合现实的光场三维显示研究综述[J].计算机辅助设计与图形学学报，2016，28（6）：905-912.
SU C，LI H F.Survey on light field 3D display applied to mixed reality[J].Journal of Computer-Aided Design and Computer Graphics，2016，28（6）：905-912.
[9] 陈芳，赵喆，张道强，等.特征重加权U-Net超声图像分割的骨结构重建与增强现实显示[J].计算机辅助设计与图形学学报，2021，33（3）：448-456.
CHEN F，ZHAO Z，ZHANG D Q，et al.3D reconstruction and augmented reality display of bone structure based on ultrasound segmentation with feature-reweighted U-Net[J].Journal of Computer-Aided Design and Computer Graphics，2021，33（3）：448-456.
[10] YU Z Q，ZHANG Q B，TAO X，et al.High-performance full-color imaging system based on end-to-end joint optimization of computer-generated holography and metalens[J].Optics Express，2022，30：40871-40883.
[11] KIM Y，HONG K，YEOM H J，et al.Wide-viewing holographic stereogram based on self-interference incoherent digital holography[J].Optics Express，2022，30（8）：12760-12774.
[12] MARTINEZ C，LEE Y，CLEMENT N，et al.Multi-user volumetric 360° display based on retro-reflective transparent surfaces[J].Optics Express，2020，28：39524-39543.
[13] WANG X，HUA H.Time-multiplexed method for view density enhancement in integral imaging-based light field displays[J].Optics Letters，2022，47：4471-4474.
[14] WEI J，WANG S，ZHAO Y，et al.Synthetic aperture integral imaging using edge depth maps of unstructured monocular video[J].Optics Express，2018，26：34894-34908.
[15] CHOI H M，HWANG Y S，KIM E S.Field-of-view enhanced integral imaging with dual prism arrays based on perspective-dependent pixel mapping[J].Optics Express，2022，30：11046-11065.
[16] MARTINEZ-CORRAL M，DORADO A，BARREIRO J C，et al.Recent advances in the capture and display of macroscopic and microscopic 3D scenes by integral imaging[J].Proceedings of the IEEE，2017，105（5）：825-836.
[17] JAVIDI B，CARNICER A，ARAI J，et al.Roadmap on 3D integral imaging：sensing，processing，and display[J].Optics Express，2020，28：32266-32293.
[18] 谭立行，鲁嘉淇，张笑楠，等.基于轻量级OpenPose改进的幻影机手势交互系统[J].计算机工程与应用，2021，57（16）：159-166.
TAN L X，LU J Q，ZHANG X N，et al.Improved ghost machine gesture interaction system based on lightweight OpenPose[J].Computer Engineering and Applications，2021，57（16）：159-166.
[19] 韦健，王世刚，王学军，等.一种三维模型二维视图与裸眼三维影像双屏联动显示方法：ZL2021101613718[P].2021-12-07.
   WEI J，WANG S G，WANG X J，et al.A dual-screen linkage and display method for 2D view and naked-eye 3D image of 3D models：ZL2021101613718[P].2021-12-07.
[20] 邓欢，李述利，吴非，等.增大3D观看视角的会聚式集成成像[J].光学精密工程，2014，22（1）：1-6.
DENG H，LI S L，WU F，et al.Toed-in integral imaging with enhanced 3D viewing angle[J].Optics and Precision Engineering，2014，22（1）：1-6.
[21] WANG Q H，LI X F，ZHOU L，et al.Cross-talk reduction by correcting the subpixel position in a multiview autostereoscopic three-dimensional display based on a lenticular sheet[J].Applied Optics，2011，50（7）：B1-B5.
[22] 梅康平，张红民，李晓峰.基于VTK光线投射法的CT图像三维重建[J].计算机与数字工程，2008，36（5）：135-136.
   MEI K P，ZHANG H M，LI X F.3D reconstruction of CT image based on the ray-casting method of VTK[J].Computer and Digital Engineering，2008，36（5）：135-136.
[23] 吴东，刘星宇，张逸凌，等.人工智能辅助全髋关节置换术三维规划系统的研发及临床应用研究[J].中国修复重建外科杂志，2020，34（9）：1077-1084.
   WU D，LIU X Y，ZHANG Y L，et al.Research and application of artificial intelligence based three-dimensional preoperative planning system for total hip arthroplasty[J].Chinese Journal of Reparative and Reconstructive Survey，2020，34（9）：1077-1084.
[24] 赵玉婷，钮建伟，张人杰，等.卷积神经网络用于关节角度识别与姿势评估[J].计算机工程与应用，2019，55（16）：209-216.
   ZHAO Y T，NIU J W，ZHANG R J，et al.Convolutional neural network for joint angle recognition and posture assessment[J].Computer Engineering and Applications，2019，55（16）：209-216.