Overview of Human Behavior Recognition Based on Deep Learning

doi:10.3778/j.issn.1002-8331.2201-0096

Abstract

Abstract: Human behavior recognition aims to retrieve and identify the target behavior in surveillance video, which is a research hotspot in the field of artificial intelligence. Human behavior recognition algorithm based on traditional methods has some shortcomings, such as large dependence on sample data and easy to be affected by environmental noise. In order to solve this problem, many human behavior recognition algorithms based on deep learning are proposed for different application scenarios. Firstly, the traditional feature extraction methods and feature extraction methods based on deep learning in human behavior recognition task are introduced. Secondly, the human behavior recognition algorithms based on deep learning are summarized from two aspects of performance and application. The idea of human behavior recognition method based on 3D convolutional neural network, hybrid network, two-stream convolutional neural network and few-shot learning（FSL） and its performance on UCF101 and HMDB51 datasets are analyzed. Thirdly, on the basis of deep learning, the advantages, disadvantages and effectiveness of mainstream model migration methods are summarized. Finally, the shortcomings of existing human behavior recognition algorithms based on deep learning are summarized, and the possibility of FSL algorithm represented by meta-learning and transformer that will become the mainstream algorithm of future models is discussed. At the same time, the future development direction of human behavior recognition based on deep learning is prospected.

Key words: behavior recognition, deep learning, two-stream convolution network, few-shot learning（FSL）, meta-learning

摘要： 人体行为识别旨在对视频监控中的人体行为进行检索并识别，是人工智能领域的研究热点。基于传统方法的人体行为识别算法存在对样本数据依赖大、易受环境噪声影响等不足。为解决此问题，许多适用于不同应用场景的基于深度学习的人体行为识别算法被提出。介绍了人体行为识别任务中传统特征提取方法和基于深度学习的特征提取方法；从性能和应用两方面对基于深度学习的人体行为识别算法进行总结，重点分析了基于3D卷积神经网络、混合网络、双流卷积神经网络和少样本学习（few-shot learning，FSL）的人体行为识别方法及其在UCF101和HMDB51数据集上的表现；在深度学习的基础上，归纳了主流模型迁移方法的优缺点及其有效性；总结了现有基于深度学习的人体行为识别算法存在的不足，并讨论了以元学习（meta-learning）和transformer为代表的FSL算法将成为未来模型主流算法的可能性，同时对未来基于深度学习的人体行为识别算法的发展方向进行展望。

关键词: 行为识别, 深度学习, 双流卷积网络, 少样本学习, 元学习

DENG Miaolei, GAO Zhendong, LI Lei, CHEN Si. Overview of Human Behavior Recognition Based on Deep Learning[J]. Computer Engineering and Applications, 2022, 58(13): 14-26.

邓淼磊, 高振东, 李磊, 陈斯. 基于深度学习的人体行为识别综述[J]. 计算机工程与应用, 2022, 58(13): 14-26.

References

[1] HATIRNZA E，SAH M，DIREKOGLU C.A novel framework and concept-based semantic search Interface for abnormal crowd behaviour analysis in surveillance videos[J].Multimedia Tools & Applications，2020，79：17579-17617.
[2] 黄凯奇，陈晓棠，康运锋，等.智能视频监控技术综述[J].计算机学报，2015，38（6）：1093-1118.
HUANG K Q，CHEN X T，KANG Y F，et al.Intelligent visual surveillance：A review[J].Chinese Journal of Computers，2015，38（6）：1093-1118.
[3] QIAN Y J，GUO M L，JIN W Y，et al.A model based method of pedestrian abnormal behavior detection in traffic scene[C]/Proceedings of IEEE First International Smart Cities Conference（ISC2），2015：1-6.
[4] LENTZAS A，VRAKAS D.Non-intrusive human activity recognition and abnormal behavior detection on elderly people：A review[J].Artificial Intelligence Review，2019，53：1-47.
[5] 于乃功，柏德国.基于姿态估计的实时跌倒检测算法[J].控制与决策，2020，35（11）：2761-2766.
YU N G，BAI D G.Real-time fall detection algorithm based on pose estimation[J].Control and Decision，2020，35（11）：2761-2766.
[6] 王佳铖，鲍劲松，刘天元，等.基于工件注意力的车间作业行为在线识别方法[J].计算机集成制造系统，2021，27（4）：1099-1107.
WANG J C，BAO J S，LIU T Y，et al.Online method for worker operation recognition based on attention of workpiece[J].Computer Integrated Manufacturing Systems，2021，27（4）：1099-1107.
[7] LU J，YAN W Q，NGUYEN M.Human behaviour recognition using deep learning[C]//Proceedings of the 15th IEEE International Conference on Advanced Video and Signal Based Surveillance（AVSS），2018：1-6.
[8] VISHWAKARMA D K，KAPOOR R.Hybrid classifier based human activity recognition using the silhouette and cells[J].Expert Systems with Applications，2015，42（20）：6957-6965.
[9] MAITY S，BHATTACHARJEE D，CHAKRABARTI A.A novel approach for human action recognition from silhouette images[J].IETE Journal of Research，2017，63（2）：160-171.
[10] TANG Z，GU R，HWANG J N.Joint multi-view people tracking and pose estimation for 3D scene reconstruction[C]//Proceedings of the IEEE International Conference on Multimedia and Expo（ICME），2018：1-6.
[11] CHAI Y，SAPP B，BANSAI M，et al.Multipath：Multiple probabilistic anchor trajectory hypotheses for behavior prediction[J].arXiv：1910.05449，2019.
[12] AN L，TSOU M H，CROOK S E S，et al.Space-time analysis：Concepts，quantitative methods，and future directions[J].Annals of the Association of American Geographers，2015，105（5）：891-914.
[13] XIA L，LI Z.A new method of abnormal behavior detection using LSTM network with temporal attention mechanism[J].The Journal of Supercomputing，2021，77（4）：3223-3241.
[14] WANG Y，WANG S，ZHOU M，et al.TS-I3D based hand gesture recognition method with radar sensor[J].IEEE Access，2019，7：22902-22913.
[15] SUN L，JIA K，YEUNG D Y，et al.Human action recognition using factorized spatio-temporal convolutional networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：4597-4605.
[16] DU Y，FU Y，WANG L.Skeleton based action recognition with convolutional neural network[C]//Proceedings of the 3rd IAPR Asian Conference on Pattern Recognition（ACPR），2015：579-583.
[17] KAMAL S，JALAL A，KIM D.Depth images-based human detection，tracking and activity recognition using spatiotemporal features and modified HMM[J].Journal of Electrical Engineering and Technology，2016，11（6）：1857-1862.
[18] SI C，CHEN W，WANG W，et al.An attention enhanced graph convolutional LSTM network for skeleton-based action recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：1227-1236.
[19] 苏江毅，宋晓宁，吴小俊，等.多模态轻量级图卷积人体骨架行为识别方法[J].计算机科学与探索，2021，15（4）：733-742.
SU J Y，SONG X N，WU X J，et al.Skeleton based action recognition algorithm on multi-modal lightweight graph convolutional network[J].Journal of Frontiers of Computer Science and Technology，2021，15（4）：733-742.
[20] ZHANG K，LING W.Joint motion information extraction and human behavior recognition in video based on deep learning[J].IEEE Sensors Journal，2019，20：11919-11926.
[21] ZHAO Z，ZOU W，WANG J J.Action recognition based on C3D network and adaptive keyframe extraction[C]//Proceedings of the IEEE 6th International Conference on Computer and Communications（ICCC），2020：2441-2447.
[22] LIU G，ZHANG C，XU Q，et al.I3D-shufflenet based human action recognition[J].Algorithms，2020，13（11）：301.
[23] YAN L C，YOSHUA B，GEOFFREY H.Deep learning[J].Nature，2015，521：436-444.
[24] ZEILER M D，FERGUS R.Visualizing and understanding convolutional networks[C]//Proceedings of the European Conference on Computer Vision，2014：818-833.
[25] SIMONYAN K，VEDALDI A，ZISSERMAN A.Deep inside convolutional networks：Visualising image classification models and saliency maps[J].arXiv：1312.6034，2013.
[26] BALLESTER P，ARAUJO R M.On the performance of GoogLeNet and AlexNet applied to sketches[C]//Proceedings of the 30th AAAI Conference on Artificial Intelligence，2016.
[27] TARG S，ALMEIDA D，LYMAN K.Resnet in resnet：Generalizing residual architectures[J].arXiv：1603.08029，2016.
[28] NGUYEN N C，VUDINH T.A new architecture for tele-radiology networks[C]//Proceedings of the International Conference on Advanced Technologies for Communications（ATC），2015：394-399.
[29] KIM W，CHOI H K，JANG B T，et al.Driver distraction detection using single convolutional neural network[C]//Proceedings of the International Conference on Information and Communication Technology Convergence（ICTC），2017：1203-1205.
[30] SANDLER M，HOWARD A，ZHU M，et al.Mobilenetv2：Inverted residuals and linear bottlenecks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：4510-4520.
[31] HOWARD A，SANDLER M，CHU G，et al.Searching for mobilenetv3[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：1314-1324.
[32] JAIN M，GEMERT J V，SNOEK C G M.University of Amsterdam at THUMOS challenge 2014[M]//THUMOS challenge 2014：notebook papers.Orlando，FL：Center for Research in Computer Vision，University of Central Florida，2014.
[33] WANG L，QIAO Y，TANG X.Action recognition and detection by combining motion and appearance features[J].THUMOS14 Action Recognition Challenge，2014，1（2）：2.
[34] JI S，XU W，YANG M，et al.3D convolutional neural networks for human action recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2012，35（1）：221-231.
[35] CARREIRA J，ZISSERMAN A.Quo Vadis，action recognition? a new model and the kinetics dataset[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：6299-6308.
[36] TRAN D，RAY J，SHOU Z，et al.Convnet architecture search for spatiotemporal feature learning[J].arXiv：1708. 05038，2017.
[37] LIU K，LIU W，GAN C，et al.T-C3D：Temporal convolutional 3D network for real-time action recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2018：7138-7145.
[38] LU B，LV Z，ZHU S.Pseudo-3D residual networks based anomaly detection in surveillance videos[C]//Proceedings of the Chinese Automation Congress（CAC），2019：3769-3773.
[39] TRAN D，WANG H，TORRESANI L，et al.A closer look at spatiotemporal convolutions for action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：6450-6459.
[40] PENG B，YAO Z，WU Q，et al.3D Convolutional neural network for human behavior analysis in intelligent sensor network[J].Mobile Networks and Applications，2022，80：1-10.
[41] HU Z，HU Y，LIU J，et al.3D separable convolutional neural network for dynamic hand gesture recognition[J].Neurocomputing，2018，318：151-161.
[42] QIU Z，YAO T，MEI T.Learning spatio-temporal representation with pseudo-3D residual networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：5533-5541.
[43] WANG M，ZHU Y，SUN Z，et al.Abnormal behavior detection of ATM surveillance videos based on pseudo-3D residual network[C]//Proceedings of the IEEE 4th International Conference on Cloud Computing and Big Data Analysis（ICCCBDA），2019：412-417.
[44] 王新文，谢林柏，彭力.跌倒异常行为的双重残差网络识别方法[J].计算机科学与探索，2020，14（9）：1580-1589.
WANG X W，XIE L B，PENG L.Double residual network recognition method for falling abnormal behavior[J].Journal of Frontiers of Computer Science and Technology，2020，14（9）：1580-1589.
[45] JIANG H， PAN Y， ZHANG J，et al.Battlefield target aggregation behavior recognition model based on multi-scale feature fusion[J].Symmetry，2019，11（6）：761.
[46] SHEN H X，LI Y，CHAEN H，et al.Research on human action recognition based on improved pooling Algorithm[C]//Proceedings of the Chinese Control And Decision Conference（CCDC），2020：3306-3310.
[47] YU G，LIU J，ZHANG C.An abnormal behavior recognition method based on fusion features[C]//Proceedings of the International Conference on Intelligent Robotics and Applications，2021：222-232.
[48] TSENG C K，LIAO C C，SHEN P C，et al.Using C3D to detect rear overtaking behavior[C]//Proceedings of the IEEE International Conference on Image Processing（ICIP），2019：151-154.
[49] ZHAO C，CHEN M，ZHAO J，et al.3D behavior recognition based on multi-modal deep space-time learning[J].Applied Sciences，2019，9（4）：716.
[50] YU S，CHENG Y，XIE L，et al.A novel recurrent hybrid network for feature fusion in action recognition[J].Journal of Visual Communication and Image Representation，2017，49：192-203.
[51] LI S，ZHAO Z，SU F.A spatio-temporal hybrid network for action recognition[C]//Proceedings of the IEEE Visual Communications and Image Processing（VC） IP，2019：1-4.
[52] LI Z，GAVRILYUK K，GAVVES E，et al.VideoLSTM convolves，attends and flows for action recognition[J].Computer Vision and Image Understanding，2018，166：41-50.
[53] TANBERK S，KILIMCI Z H，TUKEL D B，et al.A hybrid deep model using deep learning and dense optical flow approaches for human activity recognition[J].IEEE Access，2020，8：19799-19809.
[54] JIN W，XUE Y，MENG X，et al.Research on behavior recognition algorithm based on SE-I3D-GRU network[J].High Technology Letters，2021，27（2）：163-172.
[55] JIANG Y G，WU Z，TANG J，et al.Modeling multimodal clues in a hybrid deep learning framework for video classification[J].IEEE Transactions on Multimedia，2018，20（11）：3137-3147.
[56] ALAZZAWI N A.Human action recognition based on hybrid deep learning model and shearlet transform[C]//Proceedings of the 12th International Conference on Information Technology and Electrical Engineering（ICITEE），2020：152-155.
[57] JAOUEDI N，BOUJNAH N，BOUHLEL M S.A new hybrid deep learning model for human action recognition[J].Journal of King Saud University-Computer and Information Sciences，2020，32（4）：447-453.
[58] 钱慧芳，易剑平，付云虎.基于深度学习的人体动作识别综述[J].计算机科学与探索，2021，15（3）：438-455.
QIAN H F，YI J P，FU Y H.Review of human action recognition based on deep learning[J].Journal of Frontiers of Computer Science and Technology，2021，15（3）：438-455.
[59] ULLAH H，KHAN S D，ULLAH M，et al.Two stream model for crowd video classification[C]//Proceedings of the 8th European Workshop on Visual Information Processing（EUVIP），2019：93-98.
[60] WANG L，XIONG Y，WANG Z，et al.Temporal segment networks for action recognition in videos[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2018，41（11）：2740-2755.
[61] YE Q，LIANG Z，ZHONG H，et al.Human behavior recognition based on time correlation sampling two-stream heterogeneous grafting network[J].Optik，2022，251：168402.
[62] FEICHTENHOFER C，PINZ A，ZISSERMAN A.Convolutional two-stream network fusion for video action recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：1933-1941.
[63] WANG C H，WEI Y Q，GUO D，et al.Human behavior recognition under occlusion based on two-stream network combined with BiLSTM[C]//Proceedings of the Chinese Control and Decision Conference，2020：3311-3316.
[64] 张红颖，安征.基于改进双流时空网络的人体行为识别[J].光学精密工程，2021，29（2）：420-429.
ZHANG H Y，AN Z.Human action recognition based on improved two-stream spatiotemporal network[J].Optics and Precision Engineering，2021，29（2）：420-429.
[65] LIU C，YING J，YANG H，et al.Improved human action recognition approach based on two-stream convolutional neural network model[J].The Visual Computer，2021，37（6）：1327-1341.
[66] SARABU A，SANTRA A K.Distinct two-stream convolutional networks for human action recognition in videos using segment?based temporal modeling[J].Data，2020，5（4）：104.
[67] YANG X，LIU L，WANG N，et al.A two-stream dynamic pyramid representation model for video-based person re-identification[J].IEEE Transactions on Image Processing，2021，30：6266-6276.
[68] TANG Y，WANG Y，XU Y，et al.Beyond dropout：Feature map distortion to regularize deep neural networks[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2020：5964-5971.
[69] ZUNINO A，BARGAL S A，MORERIO P，et al.Excitation dropout：Encouraging plasticity in deep neural networks[J].International Journal of Computer Vision，2021，129（4）：1139-1152.
[70] DAI C，LIU X，LAI J.Human action recognition using two-stream attention based LSTM networks[J].Applied Soft Computing，2020，86：105820.
[71] CHI L，TIAN G，MU Y，et al.Two-stream video classification with cross-modality attention[C]//Proceedings of the IEEE International Conference on Computer Vision Workshop（ICCVW），2019：4511-4520.
[72] PENG Y，ZHAO Y，ZHANG J.Two-stream collaborative learning with spatial-temporal attention for video classification[J].IEEE Transactions on Circuits and Systems for Video Technology，2018，29（3）：773-786.
[73] KUMAR N，NARANG S.Few shot activity recognition using variational inference[J].arXiv：2108.08990，2021.
[74] LIU J，SONG L，QIN Y.Prototype rectification for few-shot learning[C]//Proceedings of the European Conference on Computer Vision，2020：741-756.
[75] SUN C，BARADEL F，MURPHY K，et al.Learning video representations using contrastive bidirectional transformer[J].arXiv：1906.05743，2019.
[76] ZOU Y，ZHANG S，CHEN K，et al.Compositional few-shot recognition with primitive discovery and enhancing[C]//Proceedings of the 28th ACM International Conference on Multimedia，2020：156-164.
[77] LI X，HE Y，ZHANG J A，et al.Supervised domain adaptation for few-shot radar-based human activity recognition[J].IEEE Sensors Journal，2021，21（22）：25880-25890.
[78] LI S，LIU H，QIAN R，et al.TA2N：Two-stage action alignment network for few-shot action recognition[J].arXiv：2107.04782，2021.
[79] LIU X，JI Z，PANG Y，et al.DGIG-Net：Dynamic graph-in-graph networks for few-shot human-object interaction[J].IEEE Transactions on Cybernetics，2021（1）：1-13.
[80] RIZVE M N，KHAN S，KHAN F S，et al.Exploring complementary strengths of invariant and equivariant representations for few-shot learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：10836-10846.
[81] KANG D，KWON H，MIN J，et al.Relational embedding for few-shot classification[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：8822-8833.
[82] HAN G，HE Y，HUANG S，et al.Query adaptive few-shot object detection with heterogeneous graph convolutional networks[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：3263-3272.
[83] FU Y，ZHANG L，WANG J，et al.Depth guided adaptive meta-fusion network for few-shot video recognition[C]//Proceedings of the 28th ACM International Conference on Multimedia，2020：1142-1151.
[84] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems，2017：5998-6008.
[85] YANG P，METTES P，SNOEK C G M.Few-shot transformation of common actions into time and Space[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：16031-16040.
[86] FAN R，CHU W，CHANG P，et al.An improved single step non-autoregressive transformer for automatic speech recognition[J].arXiv：2106.09885，2021.
[87] MAZZIA V，ANGARANO S，SALVETTI F，et al.Action transformer：A self-attention model for short-time human action recognition[J].arXiv：2107.00606，2021.
[88] BOUGHIDA A，KOUAHLA M N，LAFIFI Y.A novel approach for facial expression recognition based on gabor filters and genetic algorithm[J].Evolving Systems，2022，13：331-345.
[89] BYEON Y H，KIM D，LEE J，et al.Body and hand-object ROI-based behavior recognition using deep learning[J].Sensors，2021，21（5）：1838.
[90] 周波，李俊峰.基于多流卷积神经网络的行为识别[J].计算机系统应用，2021，30（8）：118-125.
ZHOU B，LI J F.Behavior recognition based on multi-stream convolutional neural network[J].Computer Systems & Applications，2021，30（8）：118-125.
[91] HSUEH Y L， LIE W N， GUO G Y.Human behavior recognition from multiview videos[J].Information Sciences，2020，517：275-296.
[92] LEE E J，KO B C，NAM J Y.Recognizing pedestrian’s unsafe behaviors in far-infrared imagery at night[J].Infrared Physics & Technology，2016，76：261-270.
[93] 何坚，郭泽龙，刘乐园，等.基于滑动窗口和卷积神经网络的可穿戴人体活动识别技术[J].电子与信息学报，2022，44（1）：168-177.
HE J，GUO Z L，LIU L Y，et al.Human activity recognition technology based on sliding window and convolutional neural network[J].Journal of Electronics & Information Technology，2022，44（1）：168-177.
[94] XIE T T，TZELEPIS C，FU F，et al.Few-shot action localization without knowing boundaries[J].arXiv：2106. 04150，2021.
[95] KUJANI T，KUMAR V D.Head movements for behavior recognition from real time video based on deep learning ConvNet transfer learning[J].Journal of Ambient Intelligence and Humanized Computing，2021（1）：1-15.
[96] RESCIGNO M，SPEZIALETTI M，ROSSI S.Personalized models for facial emotion recognition through transfer learning[J].Multimedia Tools and Applications，2020，79：35811-35828.
[97] YANG X，ZHANG Y，LV W，et al.Image recognition of wind turbine blade damage based on a deep learning model with transfer learning and an ensemble learning classifier[J].Renewable Energy，2021，163：386-397.
[98] BU Q，YANG G，MING X，et al.Deep transfer learning for gesture recognition with WiFi signals[J].Personal and Ubiquitous Computing，2020，939：1-12.
[99] SOOMRO K，ZAMIR A R，SHAH M.UCF101：A dataset of 101 human actions classes from videos in the wild[J].arXiv：1212.0402，2012.
[100] KUEHNE H，JHUANG H，Garrote E，et al.HMDB：A large video database for human motion recognition[C]//Proceedings of the International Conference on Computer Vision，2011：2556-2563.
[101] PENG X，WANG L，WANG X，et al.Bag of visual words and fusion methods for action recognition：Comprehensive study and good practice[J].Computer Vision and Image Understanding，2016，150：109-125.
[102] LI B，LI X，ZHANG Z，et al.Spatio-temporal graph routing for skeleton-based action recognition[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2019：8561-8568.
[103] FU Z，HE X，WANG E，et al.Personalized human activity recognition based on integrated wearable sensor and transfer learning[J].Sensors，2021，21（3）：885.