Progress of Instantiated Reality Augmentation Method for Smart Phone Indoor Scene Elements

doi:10.3778/j.issn.1002-8331.2309-0376

Abstract

Abstract: Indoor mobile phone navigation and location services are current research hotspots, of which scene element instantiation reality augmentation methods are an important part. Instantiated segmentation is a challenging and fundamental task in scene element perception, and augmented reality is an effective way to apply digital twin building maps, both of which are of great importance in the field of indoor location navigation. At present, augmented reality technology is mainly applied to semantic enhancements in scenes, and AR enhancements for smartphone indoor navigation only stay in the visual visualization effect, and have not yet really penetrated to the level of enhancement of elemental instances in scenes. To address this problem, this paper proposes an AR research idea of mobile phone scene element instantiation, by identifying objects in indoor scenes and matching them with building maps, the corresponding stored element information in the building maps will be enhanced and displayed using AR technology, thus assisting pedestrians in indoor navigation and location services and other related applications, and improving the information level of location services such as indoor positioning and navigation for users. This paper provides a systematic overview of instance segmentation and augmented reality methods for smartphone-side video, and analyses the characteristics and applicable scenarios of the relevant methods, summaries the research progress of instance segmentation and augmented reality in mobile, and finally discusses the application prospects of instantiated reality augmentation methods for indoor scene elements in the field of navigation and location services.

Key words: augmented reality, instance segmentation, deep learning, mobile indoor location-based navigation, building map matching

摘要： 室内手机导航与位置服务是当前的研究热点，场景要素实例化现实增强方法是其中重要的组成部分。实例分割是场景要素感知中一项具有挑战性的基本任务，增强现实是数字孪生建筑物地图应用的有效途径，两者在室内定位导航领域有着重要意义。当前，增强现实技术主要应用在对场景中的语义增强，智能手机室内导航AR增强也只是停留在视觉可视化效果方面，尚没有真正深入到对场景中要素实例的增强层面。针对该问题，提出手机场景要素实例化AR研究思路，通过识别室内场景中的对象并与建筑物地图进行匹配，将建筑物地图中对应存储的要素信息利用AR技术进行增强显示，进而辅助行人进行室内导航与位置服务等相关应用，提升用户室内定位导航等位置服务的信息化水平。对智能手机端视频的实例分割和增强现实方法进行了系统的梳理，并分析了相关方法的特点和适用场景，总结了移动端实例分割和增强现实的研究进展，最后探讨了室内场景要素实例化现实增强方法在导航与位置服务领域的应用前景。

关键词: 增强现实, 实例分割, 深度学习, 手机室内定位导航, 建筑物地图匹配

LIU Jianhua, WANG Nan, BAI Mingchen. Progress of Instantiated Reality Augmentation Method for Smart Phone Indoor Scene Elements[J]. Computer Engineering and Applications, 2024, 60(7): 58-69.

刘建华, 王楠, 白明辰. 手机室内场景要素实例化现实增强方法研究进展[J]. 计算机工程与应用, 2024, 60(7): 58-69.

References

[1] 徐舒婷, 郑先伟, 谢潇, 等. 面向虚实融合的单体建筑物实时识别与定位[J]. 武汉大学学报 (信息科学版), 2023, 48(4): 542-549.
XU S T, ZHENG X W, XIE X, et al. Real-time building instance recognition for vector map and real scene fusion[J]. Geomatics and Information Science of Wuhan University, 2023, 48(4): 542-549.
[2] ROH D, LEE J. Augmented reality-based navigation using deep learning-based pedestrian and personal mobility user recognition—a comparative evaluation for driving assistance[J]. IEEE Access, 2023, 11: 62200-62211.
[3] WANG Z. An AR map virtual-real fusion method based on element recognition[J]. ISPRS International Journal of Geo-Information, 2023, 12: 126.
[4] 陈锐志, 王磊, 李德仁, 等. 导航与遥感技术融合综述[J]. 测绘学报, 2019, 48(12): 1507-1522.
CHEN R Z, WANG L, LI D R, et al. A survey on the fusion of the navigation and the remote sensing techniques[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(12): 1507-1522.
[5] 陈锐志, 钱隆, 牛晓光, 等. 基于数据与模型双驱动的音频/惯性传感器耦合定位方法[J]. 测绘学报, 2022, 51(7): 1160-1171.
CHEN R Z, QIAN L, NIU X G, et al. Fusing acoustic ranges and inertial sensors using a data and model dual-driven approach[J]. Acta Geodaetica et Cartographica Sinica, 2022, 51(7): 1160-1171.
[6] 高翔, 安辉, 陈为, 等. 移动增强现实可视化综述 [J]. 计算机辅助设计与图形学学报, 2018, 30(1): 1-8.
GAO X, AN H, CHEN W, et al. A survey on mobile augmented reality visualization[J]. Journal of Computer-Aided Design & Computer Graphics, 2018, 30(1): 1-8.
[7] CHEN L Y, LI S B, BAI Q, et al. Review of image classification algorithms based on convolutional neural networks[J]. Remote Sensing, 2021, 13(22): 4712.
[8] XU Y S, ZHANG H Z. Convergence of deep convolutional neural networks[J]. Neural Networks, 2022, 153: 553-563.
[9] BOLYA D, ZHOU C, XIAO F, et al. YOLACT++: better real-time instance segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(2): 1108-1121.
[10] WU J, YARRAM S, LIANG H, et al. Efficient video instance segmentation via tracklet query and proposal[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022: 949-958.
[11] LI D, LI R, WANG L, et al. You only infer once: cross-modal meta-transfer for referring video object segmentation[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2022: 1297-1305.
[12] KINI J, SHAH M. Tag-based attention guided bottom-up approach for video instance segmentation[C]//Proceedings of the 2022 26th International Conference on Pattern Recognition (ICPR), 2022: 3536-3542.
[13] ZHU F, YANG Z, YU X, et al. Instance as identity: a generic online paradigm for video instance segmentation[J]. arXiv:2208.03079, 2022.
[14] GANESH P, CHEN Y, YANG Y, et al. YOLO-ReT: towards high accuracy real-time object detection on edge GPUs[C]//Proceedings of the 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021: 1311-1321.
[15] SHIN A, ISHII M, NARIHIRA T. Perspectives and prospects on Transformer architecture for cross-modal tasks with language and vision[J]. International Journal of Computer Vision, 2022, 130(2): 435-454.
[16] YAO H Y, WAN W G, LI X. End-to-end pedestrian trajectory forecasting with Transformer network[J]. ISPRS International Journal of Geo-Information, 2022, 11(1): 44.
[17] 田永林, 王雨桐, 王建功, 等. 视觉Transformer研究的关键问题: 现状及展望[J]. 自动化学报, 2022, 48(4): 957-979.
TIAN Y L, WANG Y T, WANG J G, et al. Key problems and progress of vision Transformers: the state of the art and prospects[J]. Acta Automatica Sinica, 2022, 48(4): 957-979.
[18] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, California, USA, 2017: 6000-6010.
[19] WANG Y, XU Z, WANG X, et al. End-to-end video instance segmentation with Transformers[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020: 8737-8746.
[20] ZHOU D, YU Z, XIE E, et al. Understanding the robustness in vision Transformers[J]. arXiv:2204.12451, 2022.
[21] WU J, JIANG Y, SUN P, et al. Language as queries for referring video object segmentation[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022: 4964-4974.
[22] JIN P, MOU L, XIA G S, et al. Anomaly detection in aerial videos with transformers[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-13.
[23] LI N, TU W, AI H. A sparse feature matching model using a Transformer towards large-view indoor visual localization[J]. Wireless Communications and Mobile Computing, 2022: 1243041.
[24] MEHTA S, RASTEGARI M. MobileViT: light-weight, general-purpose, and mobile-friendly vision Transformer[J]. arXiv:2110.02178, 2021.
[25] HEO B, YUN S, HAN D, et al. Rethinking spatial dimensions of vision Transformers[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021: 11916-11925.
[26] PAN J, BULAT A, TAN F, et al. EdgeViTs: competing light-weight CNNs on mobile devices with vision Transformers[C]//Proceedings of the European Conference on Computer Vision, 2022.
[27] CHEN Y, DAI X, CHEN D, et al. Mobile-former: bridging MobileNet and Transformer[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022: 5260-5269.
[28] YANG C, QIAO S, YU Q, et al. MOAT: alternating mobile convolution and attention brings strong vision models[C]//Proceedings of the International Conference on Learning Representations, 2023.
[29] TAN M, CHEN B, PANG R, et al. MnasNet: platform-aware neural architecture search for mobile[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018: 2815-2823.
[30] MEHTA S, RASTEGARI M, CASPI A, et al. ESPNet: efficient spatial pyramid of dilated convolutions for semantic segmentation[C]//Proceedings of the European Conference on Computer Vision (ECCV 2018), Cham, 2018: 561-580.
[31] MEHTA S, RASTEGARI M, SHAPIRO L G, et al. ESPNetv2: a light-weight, power efficient, and general purpose convolutional neural network[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018: 9182-9192.
[32] MAAZ M, SHAKER A M, CHOLAKKAL H, et al. EdgeNeXt: efficiently amalgamated CNN-Transformer architecture for mobile vision applications[C]//Proceedings of the European Conference on Computer Vision, 2022.
[33] HOWARD A G, ZHU M, CHEN B, et al. MobileNets: efficient convolutional neural networks for mobile vision applications[J]. arXiv:1704.04861, 2017.
[34] SANDLER M, HOWARD A G, ZHU M, et al. MobileNetV2: inverted residuals and linear bottlenecks[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 4510-4520.
[35] HOWARD A G, SANDLER M, CHU G, et al. Searching for MobileNetV3[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 2019: 1314-1324.
[36] MEHTA S, RASTEGARI M. Separable self-attention for mobile vision Transformers[J]. arXiv:2206.02680, 2022.
[37] MA L. Application of AR in 3D model[C]//Proceedings of the 2021 2nd International Conference on Control, Robotics and Intelligent System, Qingdao, China, 2021: 261-265.
[38] ZHOU Y, SUN B, QI Y, et al. Mobile AR/VR in 5G based on convergence of communication and computing[J]. Telecommunications Science, 2018, 34(8): 19-33.
[39] LI R P, ZHAO Z F, ZHOU X, et al. Intelligent 5G: when cellular networks meet artificial intelligence[J]. IEEE Wireless Communications, 2017, 24(5): 175-183.
[40] GHASEMI Y, JEONG H, CHOI S H, et al. Deep learning-based object detection in augmented reality: a systematic review[J]. Computers in Industry, 2022, 139: 103661.
[41] HWANG S, LEE J, KANG S. Enabling product recognition and tracking based on text detection for mobile augmented reality[J]. IEEE Access, 2022, 10: 98769-98782.
[42] ZHOU B, GUVEN S. Fine-grained visual recognition in mobile augmented reality for technical support[J]. IEEE Transactions on Visualization and Computer Graphics, 2020, 26(12): 3514-3523.
[43] 王巍, 王志强, 赵继军, 等. 基于移动平台的增强现实研究[J]. 计算机科学, 2015, 42(Z11): 510-519.
WANG W, WANG Z Q, ZHAO J J, et al. Research of augmented reality based on mobile platform[J]. Computer Science, 2015, 42(Z11): 510-519.
[44] LE H, NGUYEN M, YAN W Q, et al. Augmented reality and machine learning incorporation using YOLOv3 and ARKit[J]. Applied Sciences-Basel, 2021, 11(13): 6006.
[45] LO VALVO A, CROCE D, GARLISI D, et al. A navigation and augmented reality system for visually impaired people [J]. Sensors, 2021, 21(9): 3061.
[46] REAL S, ARAUJO A. VES: a mixed-reality system to assist multisensory spatial perception and cognition for blind and visually impaired people[J]. Applied Sciences-Basel, 2020, 10(2): 523.
[47] ZHANG X C, YAO X Y, ZHU Y, et al. An ARCore based user centric assistive navigation system for visually impaired people[J]. Applied Sciences-Basel, 2019, 9(5): 989.
[48] VARELAS T, PENTEFOUNDAS A, GEORGIADIS C, et al. An AR indoor positioning system based on anchors[J]. MATTER: International Journal of Science and Technology, 2020, 6: 43-57.
[49] LU F, ZHOU H, GUO L, et al. An ARCore-based augmented reality campus navigation system[J]. Applied Sciences, 2021, 11(16): 7515.
[50] MARTIN A, CHERIYAN J, GANESH J J, et al. Indoor navigation using augmented reality[J]. EAI Endorsed Transactions on Creative Technologies, 2021: 168718.
[51] HUANG B C, HSU J, CHU E T, et al. ARBIN: augmented reality based indoor navigation system[J]. Sensors (Basel, Switzerland), 2020, 20(20): 5890.
[52] ZHOU B, GU Z, MA W, et al. Integrated BLE and PDR indoor localization for geo-visualization mobile augmented reality[C]//Proceedings of the 2020 16th International Conference on Control, Automation, Robotics and Vision (ICARCV), 2020: 1347-1353.
[53] MAHAPATRA T, TSIAMITROS N, ROHR A, et al. Pedestrian augmented reality navigator[J]. Sensors, 2023, 23: 1816.
[54] SHARIN N A, NOROWI N, ABDULLAH L, et al. GoMap: combining step counting technique with augmented reality for a mobile-based indoor map locator[J]. Indonesian Journal of Electrical Engineering and Computer Science, 2023, 29: 1792.
[55] LI X, TIAN Y, ZHANG F, et al. Object detection in the context of mobile augmented reality[C]//Proceedings of the 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2020: 156-163.
[56] KAUL O, BEHRENS K, ROHS M. Mobile recognition and tracking of objects in the environment through augmented reality and 3D audio cues for people with visual impairments[C]//Proceedings of the CHI Conference on Human Factors in Computing Systems, 2021: 1-7.
[57] CHEN J, ZHU Z. Real-time 3D object detection, recognition and presentation using a mobile device for assistive navigation[J]. SN Computer Science, 2023, 4: 543.
[58] HSIEH C C, CHEN H M, WANG S K. On-site visual construction management system based on the integration of SLAM-based AR and BIM on a handheld device[J]. KSCE Journal of Civil Engineering, 2023, 27: 4688-4707.
[59] VERMA P, AGRAWAL K, SARASVATHI V. Indoor navigation using augmented reality[C]//Proceedings of the 2020 4th International Conference on Virtual and Augmented Reality Simulations, 2020.
[60] VERYKOKOU S, BOUTSI A M, IOANNIDIS C. Mobile augmented reality for low-end devices based on planar surface recognition and optimized vertex data rendering[J]. Applied Sciences, 2021, 11(18): 8750.
[61] WANG Q, XIE Z. ARIAS: an AR-based interactive advertising system[J]. PLoS One, 2023, 18: e0285838.
[62] TSUBOKI Y, KAWAKAMI T, MATSUMOTO S, et al. A real-time background replacement system based on estimated depth for AR applications[J]. Journal of Information Processing, 2023, 31: 758-765.
[63] BARUCH G, CHEN Z, DEHGHAN A, et al. ARKitScenes-a diverse real-world dataset for 3D indoor scene understanding using mobile RGB-D data [J]. arXiv:2111.08897, 2021.
[64] FEIGL T, PORADA A, STEINER S, et al. Localization limitations of ARCore, ARKit, and hololens in dynamic large-scale industry environments[C]//Proceedings of the 15th International Conference on Computer Graphics Theory and Applications, 2020: 307-318.
[65] LIU Z, LAN G, STOJKOVIC J, et al. CollabAR: edge-assisted collaborative image recognition for mobile augmented reality[C]//Proceedings of the 2020 19th ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN), 2020: 301-312.
[66] ZHANG W, LIN S, BIJARBOONEH F H, et al. EdgeXAR: a 6-DoF camera multi-target interaction framework for MAR with user-friendly latency compensation[C]//Proceedings of the ACM on Human-Computer Interaction, 2021: 1-24.
[67] XIAO Y, AI T, YANG M, et al. A multi-scale representation of point-of-interest (POI) features in indoor map visualization[J]. International Journal of Geo-Information, 2020, 9(4): 239.
[68] 李德仁. 论可量测实景影像的概念与应用——从4D产品到5D产品 [J]. 测绘科学, 2007, 32(4): 5-7.
LI D R. On concept and application of digital measurable images-from 4D production to 5D production[J]. Science of Surveying and Mapping, 2007, 32(4): 5-7.
[69] 朱欣焰, 周成虎, 呙维, 等. 全息位置地图概念内涵及其关键技术初探[J]. 武汉大学学报(信息科学版), 2015, 40(3): 285-295.
ZHU X Y, ZHOU C H, GUO W, et al. Preliminary study on conception and key technologies of the location-based pan-information map[J]. Geomatics and Information Science of Wuhan University, 2015, 40(3): 285-295.
[70] 闾国年, 袁林旺, 俞肇元. 地理学视角下测绘地理信息再透视[J]. 测绘学报, 2017, 46(10): 1549-1556.
LV G N, YUAN L W, YU Z Y. Surveying and mapping geographical information from the perspective of geography[J]. Acta Geodaetica et Cartographica Sinica, 2017, 46(10): 1549-1556.
[71] ZHU J, WU P, LEI X. IFC-graph for facilitating building information access and query[J]. Automation in Construction, 2023, 148: 104778.
[72] LIU L, LI B, ZLATANOVA S, et al. Indoor navigation supported by the industry foundation classes (IFC): a survey [J]. Automation in Construction, 2021, 121: 103436.
[73] WEI Z, LI X, HE Z. Semantic urban vegetation modelling based on an extended CityGML description[C]//Proceedings of the 2022 Digital Landscape Architecture Conference, 2022.
[74] TANG L, YING S, LI L, et al. An application-driven LOD modeling paradigm for 3D building models[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 161: 194-207.
[75] DIAKITé A, DíAZ-VILARI?O L, BILJECKI F, et al. IFC2INDOORGML: an open-source tool for generating IndoorGML from IFC[J]. The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2022: 295-301.
[76] 罗竟妍. 建筑物实景全息地图模型构建方法研究[D]. 北京: 北京建筑大学, 2021.
LUO J Y. Research on the construction method of realistic holographic map model of building[D]. Beijing: Beijing University of Civil Engineering and Architecture, 2021.
[77] WANG Q, YE L, YUN L, et al. Pedestrian walking distance estimation based on smartphone mode recognition[J]. Remote Sensing, 2019, 11: 1140.
[78] WU Y, CHEN P, GU F, et al. HTrack : an efficient heading-aided map matching for indoor localization and tracking[J]. IEEE Sensors Journal, 2019, 19(8): 3100-3110.
[79] JIANHUA L, GUOQIANG F, JINGYAN L, et al. Mobile phone indoor scene features recognition localization method based on semantic constraint of building map location anchor[J]. Open Geosciences, 2022, 14: 1268-1289.
[80] 刘建华. 手机室内导航与位置服务[M]. ResearchGate, 2022: 69-79.
LIU J H. Mobile indoor navigation and location services[M]. ResearchGate, 2022: 69-79.
[81] 于娟, 杨琼, 鲁剑锋, 等. 高级地图匹配算法: 研究现状和趋势[J]. 电子学报, 2021, 49(9): 1818-1829.
YU J, YANG Q, LU J F, et al. Advanced map matching algorithms: a survey and trends[J]. Acta Electronica Sinica, 2021, 49(9): 1818-1829.
[82] JIANG L, CHEN C, CHEN C. L2MM: learning to map matching with deep models for low-quality GPS trajectory data[J]. ACM Transactions on Knowledge Discovery from Data, 2022, 17: 1-25.
[83] 郑诗晨, 盛业华, 吕海洋. 基于粒子滤波的行车轨迹路网匹配方法[J]. 地球信息科学学报, 2020, 22(11): 2109-2117.
ZHENG S C, SHENG Y H, LV H Y. Vehicle trajectory-map matching based on particle filter[J]. Journal of Geo-information Science, 2020, 22(11): 2109-2117.
[84] OBRADOVIC D, LENZ H, SCHUPFNER M. Fusion of map and sensor data in a modern car navigation system[J]. Journal of VLSI Signal Processing Systems for Signal Image & Video Technology, 2006, 45(1/2): 111-122.
[85] 毛江云, 吴昊, 孙未未. 路网空间下基于马尔可夫决策过程的异常车辆轨迹检测算法[J]. 计算机学报, 2018, 41(8): 1928-1942.
MAO J Y, WU H, SUN W W. Vehicle trajectory anomaly detection in road network via Markov decision process[J]. Chinese Journal of Computers, 2018, 41(8): 1928-1942.
[86] CUI G, BIAN W, WANG X. Hidden Markov map matching based on trajectory segmentation with heading homogeneity[J]. GeoInformatica, 2021, 25(1): 179-206.
[87] HARDER D, SHOUSHTARI H, STERNBERG H. Real-time map matching with a backtracking particle filter using geospatial analysis[J]. Sensors (Basel, Switzerland), 2022, 22(9): 3289.
[88] GUO G, YAN K, LIU Z, et al. Virtual wireless device-constrained robust extended Kalman filters for smartphone positioning in indoor corridor environment[J]. IEEE Sensors Journal, 2023, 23(3): 2815-2822.
[89] FENG J, LI Y, ZHAO K, et al. DeepMM: deep learning based map matching with data augmentation[J]. IEEE Transactions on Mobile Computing, 2022, 21(7): 2372-2384.
[90] HONG Y, ZHANG Y, SCHINDLER K, et al. Context-aware multi-head self-attentional neural network model for next location prediction [J]. arXiv:2212.01953, 2022.
[91] HONG Y, MARTIN H, RAUBAL M. How do you go where? Improving next location prediction by learning travel mode information using transformers[C]//Proceedings of the 30th International Conference on Advances in Geographic Information Systems, 2022: 1-10.
[92] LI Q, CAO R, ZHU J, et al. Learn then match: a fast coarse-to-fine depth image-based indoor localization framework for dark environments via deep learning and keypoint-based geometry alignment[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2023, 195: 169-177.