Modeling and Recognition Method of Elevator Passenger Abnormal Behavior Based on Digital Twin

doi:10.3778/j.issn.1002-8331.2211-0405

Abstract

Abstract: In order to solve the problems of abnormal cases lack and training samples scarcity in abnormal behavior recognition research, taking the abnormal behavior monitoring of vertical elevator passengers as the mission requirement, this paper puts forward the digital twin-based elevator passenger abnormal behavior modeling and recognition method.Through building the digital twin system architecture for elevator passenger behavior monitoring, the virtual and real mapping of elevator operation state and passenger behavior is completed.The abnormal behavior case of elevator passengers is built and the twin data of abnormal behavior is expanded based on the digital twin scene and the human body behavior modeling theory.Finally, the improved OpenPose is used to obtain skeletal features and the PCA-DNN is used to train the classification model to realize the abnormal passenger behavior fast recognition with fused twin data.Experimental results show that the proposed method not only reveals the convenience, efficiency and safety to create anomalous data with digital twin technology, but also verifies the feasibility, reliability and accuracy to train models with twin data.

Key words: elevator passenger abnormal behavior, digital twins, behavior modeling, data augmentation, skeleton detection, gesture recognition

摘要： 为了解决异常行为识别研究中存在的异常案例缺乏、训练样本稀缺问题，以垂直电梯的乘客异常行为监测为任务需求，提出了基于数字孪生的电梯乘客异常行为建模与识别方法。通过搭建电梯乘客行为监测数字孪生系统架构，完成了电梯运行状态与乘客行为的虚实映射。基于数字孪生场景与人体行为建模理论构建了电梯乘客异常行为案例，扩充了异常行为的孪生数据。利用改进的OpenPose获取骨骼特征，并使用PCA-DNN训练分类模型，实现了融合孪生数据的乘客异常行为快速识别。实验结果表明，该方法不仅展现了应用数字孪生技术创建异常数据的便捷性、高效性以及安全性，还验证了使用孪生数据进行模型训练的可行性、可靠性以及准确性。

关键词: 电梯乘客异常行为, 数字孪生, 行为建模, 数据增强, 骨骼检测, 姿态识别

LI Conglin, WANG Qibing, LU Jiawei, ZHAO Guojun, HU Hao, XIAO Gang. Modeling and Recognition Method of Elevator Passenger Abnormal Behavior Based on Digital Twin[J]. Computer Engineering and Applications, 2023, 59(19): 274-284.

李聪林, 王琪冰, 陆佳炜, 赵国军, 胡豪, 肖刚. 基于数字孪生的电梯乘客异常行为建模与识别方法[J]. 计算机工程与应用, 2023, 59(19): 274-284.

References

[1] LIU S F，AN Z L，WANG N，et al.Research on elevator passenger fall detection based on machine vision[J]. Proceedings of the IOP Conference Series：Earth and Environmental Science，2021，791（1）：012108.
[2] SUN Z，XU B L，LU M L，et al.A real-time video surveillance and state detection approach for elevator cabs[C]//Proceedings of the 2019 International Conference on Control，Automation and Information Sciences（ICCAIS），2019：1-6.
[3] ZHU Y，WANG Z.Real-time abnormal behavior detection in elevator[C]//Proceedings of the Chinese Conference on Intelligent Visual Surveillance，2016：154-161.
[4] LAN S N，JIANG S H，LI G.An elevator passenger behavior recognition method based on two-stream convolution neural network[J].Journal of Physics：Conference Series，2021，1955（1）：012089.
[5] 李佳.基于AlphaPose的电梯轿厢内乘客异常行为检测研究[D].北京：中国地质大学，2021.
LI J.Research on abnormal behavior detection of passengers in elevator car based on AlphaPose[D].Beijing：China University of Geosciences，2021.
[6] 陈龙.基于OpenPose的轿厢电梯人体行为监测系统的研究[D].云南：昆明理工大学，2021.
CHEN L.Research on human behavior monitoring system of car elevator based on OpenPose[D].Yunnan：Kunming University of Science and Technology，2021.
[7] LIU S，YIN Y，OSTADABBAS S.In-bed pose estimation：Deep learning with shallow dataset[J].IEEE Journal of Translational Engineering in Health and Medicine，2019，7（1）：1-12.
[8] ALHAMMAMI M，OOI C P，TAN W H.Violent actions against children[J].Data in Brief，2017，12（1）：480-484.
[9] KYROLLOS D G，FULLER A，GREENWOOD K，et al.Under the cover infant pose estimation using multimodal data[J].arXiv：2210.00662，2022.
[10] LIU M N，FANG S L，DONG H Y，et al.Review of digital twin about concepts，technologies，and industrial applications[J].Journal of Manufacturing Systems，2021，58（1）：346-361.
[11] 陶飞，刘蔚然，刘检华，等.数字孪生及其应用探索[J].计算机集成制造系统，2018，24（1）：1-18.
TAO F，LIU W R，LIU J H，et al.Digital twin and its potential application exploration[J].Computer Integrated Manufacturing Systems，2018，24（1）：1-18.
[12] 陶飞，刘蔚然，张萌，等.数字孪生五维模型及十大领域应用[J].计算机集成制造系统，2019，25（1）：1-18.
TAO F，LIU W R，ZHANG M，et al.Five-dimensional digital twin model and its ten applications[J].Computer Integrated Manufacturing Systems，2019，25（1）：1-18.
[13] 赵沁平，李帅，宋震，等.虚拟生理人体建模与仿真关键技术研究进展[J].中国科学基金，2022，36（2）：187-197.
ZHAO Q P，LI S，SONG Z，et al.Research progress of virtual physiological human modeling and simulation technology[J].Bulletin of National Natural Science Foundation of China，2022，36（2）：187-197.
[14] 刘庭煜，洪庆，孙毅锋，等.基于图卷积网络的数字孪生车间生产行为识别方法[J].计算机集成制造系统，2021，27（2）：501-509.
LIU T Y，HONG Q，SUN Y F，et al.Approach for recognizing production action in digital twin shop-floor based on graph convolutional network[J].Computer Integrated Manufacturing Systems，2021，27（2）：501-509.
[15] HWANG H，JANG C，PARK G，et al.Eldersim：a synthetic data generation platform for human action recognition in eldercare applications[J].arXiv：2010.14742，2020.
[16] 宋学官，何西旺，李昆鹏，等.人体骨骼数字孪生的构建方法及应用[J].机械工程学报，2022，58（18）：218-228.
SONG X G，HE X W，LI K P，et al.Construction method and application of human skeleton digital twin[J].Journal of Mechanical Engineering，2022，58（18）：218-228.
[17] XU C，HE J，ZHANG X T，et al.Geometrical kinematic modeling on human motion using method of multi-sensor fusion[J].Information Fusion，2018，41（1）：243-254.
[18] 张鋆豪，何百岳，杨旭升，等.基于可穿戴式惯性传感器的人体运动跟踪方法综述[J].自动化学报，2019，45（8）：1439-1454.
ZHANG J H，HE B Y，YANG X S，et al.A review on wearable inertial sensor based human motion tracking[J].Acta Automatica Sinica，2019，45（8）：1439-1454.
[19] CAO Z，SIMON T，WEI S E，et al.Realtime multi-person 2D pose estimation using part affinity fields[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：7291-7299.
[20] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409. 1556，2014.
[21] PHYO C N，ZIN T T，TIN P.Deep learning for recognizing human activities using motions of skeletal joints[J].IEEE Transactions on Consumer Electronics，2019，65（2）：243-252.
[22] ZHANG J B，ZHENG Y，QI D K，et al.DNN-based prediction model for spatio-temporal data[C]//Proceedings of the 24th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems，2016：1-4.
[23] 武晓栋，刘敬浩，金杰，等.基于DT及PCA的DNN入侵检测模型[J].计算机科学与探索，2021，15（8）：1450-1458.
WU X D，LIU J H，JIN J，et al.DNN intrusion detection model based on DT and PCA[J].Journal of Frontiers of Computer Science and Technology，2021，15（8）：1450-1458.
[24] AL-FAWA'REH M，AL-FAYOUMI M，NASHWAN S，et al.Cyber threat intelligence using PCA-DNN model to detect abnormal network behavior[J].Egyptian Informatics Journal，2022，23（2）：173-185.
[25] LIN T Y，MAIRE M，BELONGIE S，et al.Microsoft coco：common objects in context[C]//Proceedings of the European Conference on Computer Vision，2014：740-755.
[26] NEWELL A，HUANG Z，DENG J.Associative embedding：End-to-end learning for joint detection and grouping[C]//Advances in Neural Information Processing Systems，2017：2277-2287.
[27] IBEKWE U A，SHIRO A A.Modelling the determinants of Naira/US dollar currency exchange rates using principal component analysis（PCA） and singular value decomposition（SVD）[J].Nigerian Journal of Risk and Insurance，2022，12（1）：77-97.