Overview on Video Abnormal Behavior Detection of GAN via Human Density

doi:10.3778/j.issn.1002-8331.2110-0364

Abstract

Abstract: As an important branch of computer vision, video anomaly detection is a challenging task for intelligent video surveillance systems. It is generally referred to as automatic recognition of videos that contain abnormal targets, events or behaviors, which plays a vital role in ensuring public safety. Generative adversarial network（GAN） is anemerging unsupervised method, which can not only be used to generate images, its unique adversarial learning idea also shows good development potential in the field of anomaly detection. Firstly, the framework of the GAN is introduced. Secondly, according to the density of the scene and the object on which the action is taking place, the research status of video anomaly detection based on GAN is discussed from two aspects of individual behavior anomalies, group anomalies. These two types of abnormalities are further elaborated on the basic of reconstruction and prediction methods respectively. Thirdly, the common datasets for video anomaly detection are briefly introduced, finally, the future development is prospected.

Key words: abnormal behavior detection, generative adversarial network（GAN）, crowd anomaly detection, individual abnormal behaviors

摘要： 视频异常检测作为计算机视觉的重要分支，是智能监控系统中一项极具挑战性的任务，通常是指自动识别视频中的异常目标、行为或事件，对保障公共安全起着至关重要的作用。生成对抗网络是一种新兴的无监督方法，不仅可以用于生成图像，且其独特的对抗性学习思想在异常检测领域也显示出良好的发展潜力。介绍了生成对抗网络的框架结构；根据场景密度以及行为发生的对象，从个体行为异常、群体异常两个方面论述了生成对抗网络在视频异常检测领域的研究现状，分别基于重构和预测的方法对个体异常行为检测和群体异常行为检测作进一步阐述；简要介绍了视频异常检测的常用数据集；最后对未来发展作出了展望。

关键词: 异常行为检测, 生成对抗网络, 群体异常检测, 个体行为异常

SHEN Xulin, LI Chaobo, LI Hongjun. Overview on Video Abnormal Behavior Detection of GAN via Human Density[J]. Computer Engineering and Applications, 2022, 58(7): 21-30.

申栩林, 李超波, 李洪均. 人群密集度下GAN的视频异常行为检测进展[J]. 计算机工程与应用, 2022, 58(7): 21-30.

References

[1] 李自强，王正勇，陈洪刚，等.基于外观和动作特征双预测模型的视频异常行为检测[J].计算机应用，2021，41（10）：2997-3003.
LI Z Q，WANG Z Y，CHEN H G，et al.Video abnormal behavior detection based on dual prediction model of appearance and motion features[J].Journal of Computer Applications，2021，41（10）：2997-3003.
[2] HINTON G E，SALAKHUTDINOV R R.Reducing the dimensionality of data with neural networks[J].Science，2006，313：504-507.
[3] 苑春苗，牛瑛，郭涛，等.融合衣服特征迁移的行人重识别方法[J].计算机科学与探索，2021，15（9）：1740-1752.
YUAN C M，NIU Y，GUO T，et al.Person re-identification combined with clothing infomation transfer[J].Journal of Frontiers of Computer Science and Technology，2021，15（9）：1740-1752.
[4] 龚苏明，陈莹.时空特征金字塔模块下的视频行为识别[J/OL].计算机科学与探索：1-9（2021-03-04）[2021-10-08].http：//kns.cnki.net/kcms/detail/11.5602.TP.20210304.1631.
004.html.
GONG S M，CHEN Y.Video action recognition based on spatio-temporal feature pyramid module[J/OL].Journal of Frontiers of Computer Science and Technology：1-9 （2021-03-04）[2021-10-08].http：//kns.cnki.net/kcms/detail/11.5602.TP.20210304.1631.004.html.
[5] ULLAH W，ULLAH A，HAQ I U，et al.CNN features with bi-directional LSTM for real-time anomaly detection in surveillance networks[J].Multimedia Tools and Applications，2021，80（11）：16979-16995.
[6] MAQSOOD R，BAJWA U I，SALEEM G，et al.Anomaly recognition from surveillance videos using 3D convolution neural network[J].Multimedia Tools and Applications，2021，80（12）：18693-18716.
[7] GOODFELLOW I J，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial networks[J].Advances in Neural Information Processing Systems，2014，3：2672-2680.
[8] PANNIRSELVAM P M，GEETHA M K，KUMARAVELAN G.A comprehensive study on automated anomaly detection techniques in video surveillance[J].Annals of the Romanian Society for Cell Biology，2021，25（2）：4027-4037.
[9] NAYAK R，PATI U C，DAS S K.A comprehensive review on deep learning?based methods for video anomaly detection[J].Image and Vision Computing，2020，106：104078.
[10] 徐涛，田崇阳，刘才华.基于深度学习的人群异常行为检测综述[J].计算机科学，2021，48（9）：125-134.
XU T，TIAN C Y，LIU C H.Deep learning for abnormal crowd behavior detection：A review[J].Journal of Computer Science，2021，48（9）：125-134.
[11] SABOKROU M，KHALOOEI M，FATHY M，et al.Adversarially learned one-class classifier for novelty detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Salt Lake City，June 18-23，2018.Piscataway：IEEE，2018：3379-3388.
[12] SABOKROU M，POURREZA M，FAYYAZ M，et al.Avid：Adversarial visual irregularity detection[C]//Proceedings of the Asian Conference on Computer Vision，Perth，Dec 2-6，2018.Berlin：Springer，2018：488-505.
[13] ZAHEER M Z，LEE J，ASTRID M，et al.Old is gold：Redefining the adversarially learned one-class classifier training paradigm[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，Seattle，June 13-19，2020.Piscataway：IEEE，2020：14183-14193.
[14] SHAMA F，MECHREZ R，SHOSHAN A，et al.Adversarial feedback loop[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，Seoul，Oct 27-Nov 2，2019.Piscataway：IEEE，2019：3205-3214.
[15] VU H，TU D N，LE T，et al.Robust anomaly detection in videos using multilevel representations[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2019：5216-5223.
[16] CHEN D，YUE L，CHANG X，et al.NM-GAN：Noise-modulated generative adversarial network for video anomaly detection[J].Pattern Recognition，2021，116：107969.
[17] ATGHAEI A，ZIAEINEJAD S，RAHMATI M.Abnormal event detection in urban surveillance videos using GAN and transfer learning[J].arXiv：2011.09619，2020.
[18] SHIN W，CHO S B.CCTV image sequence generation and modeling method for video anomaly detection using generative adversarial network[C]//Proceedings of International Conference on Intelligent Data Engineering and Automated Learning，Madrid，Nov 21?23，2018.Berlin：Springer，2018：457-467.
[19] SHIN W，BU S J，CHO S B.3D-convolutional neural network with generative adversarial network and auto encoder for robust anomaly detection in video surveillance[J].International Journal of Neural Systems，2020，30（6）：2050034.
[20] LIU W，LUO W，LIAN D，et al.Future frame prediction for anomaly detection—A new baseline[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Salt Lake City，June 18-23，2018.Piscataway：IEEE，2018：6536-6545.
[21] FALK T，MAI D，BENSCH R，et al.U-Net：Deeplearning for cell counting，detection，and morphometry[J].Nature Methods，2019，16（1）：67-70.
[22] NGUYEN TN，MEUNIER J.Anomaly detection in video sequence with appearance?motion corre?spondence[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，Seoul，Oct 27-Nov 2，2019.Piscataway：IEEE，2019：1273-1283.
[23] LEI Z，DENG F，YANG X.Spatial temporal balanced generative adversarial autoencoder for anomaly detection[C]//Proceedings of the 2019 International Conference on Image，Video and Signal Processing，Shanghai，Feb 25-28，2019：1-7.
[24] LEE S，KIM H G，RO Y M.STAN：Spatio-temporal adversarial networks for abnormal event detection[C]//Proceedings of 2018 IEEE International Conference on Acoustics，Speech and Signal Processing，Calgary，April 15-20，2018.Piscataway：IEEE，2018：1323-1327.
[25] LEE S，KIM H G，RO Y M.BMAN：Bidirectional multi-scale aggregation networks for abnormal event detection[J].IEEE Transactions on Image Processing，2020，29：2395-2408.
[26] CHEN D，WANG P，YUE L，et al.Anomaly detection in surveillance video based on bidirectional prediction[J].Image and Vision Computing，2020，98：103915.
[27] TANG Y，ZHAO L，ZHANG S，et al.Integrating prediction and reconstruction for anomaly detection[J].Pattern Recognition Letters，2020，129：123-130.
[28] LIU Z，NIE Y，LONG C，et al.A hybrid video anomaly detection framework via memory-augmented flow reconstruction and flow-guided frame prediction[C]//Proceedings of the IEEE/CVF Inter-national Conference on Computer Vision，Montreal，Oct 11-17，2021.Piscataway：IEEE，2021：13588-13597.
[29] YANG M C，SHIRANI S.Multimodal generative neural network for anomaly events detection and localization in videos[C]//Proceedings of 2021 IEEE Data Science and Learning Workshop（DSLW），Toronto，June 5-6，2021.Piscataway：IEEE，2021：1-6.
[30] VU T H，BOONAERT J，AMBELLOUIS S，et al.Multi-channel generative framework and supervised learning for anomaly detection in surveillance videos[J].Sensors，2021，21（9）：3179.
[31] ZHANG Z，ZHONG S H，LIU Y.Video abnormal event detection via context cueing generative adversarial network[C]//Proceedings of 2021 IEEE International Conference on Multimedia and Expo，Shenzhen，July 5-9，2021.Piscataway：IEEE，2021：1-6.
[32] ZHANG W，WANG G，HUANG M，et al.Generative adversarial networks for abnormal event detection in videos based on self-attention mechanism[J].IEEE Access，2021，9：124847-124860.
[33] HAO Y，LI J，WANG N N，et al.Spatiotemporal consistency-enhanced network for video anomaly detection[J].Pattern Recognition，2022，121：108232.
[34] SINHA A，PADHI S，SHIKALGAR S.A survey and analysis of crowd anomaly detection techniques[C]//Proceedings of 2021 Third International Conference on Intelligent Communication Technologies and Virtual Mobile Networks，Tirunelveli，Feb 4-6，2021.Piscataway：IEEE，2021：1500-1504.
[35] RAVANBAKHSH M，NABI M，SANGINETO E，et al.Abnormal event detection in videos using generative adversarial nets[C]//Proceedings of 2017 IEEE International Conference on Image Processing，Beijing，Sep 17-20，2017.Piscataway：IEEE，2017：1577-1581.
[36] RAVANBAKHSH M，SANGINETO E，NABI M，et al.Training adversarial discriminators for cross-channel abnormal event detection in crowds[C]//Proceedings of 2019 IEEE Winter Conference on Applications of Computer Vision，Waikoloa，Jan 7-11，2019.Piscataway：IEEE，2019：1896-1904.
[37] HAN Q，WANG H，YANG L，et al.Real-time adversarial GAN-based abnormal crowd behavior detection[J].Journal of Real-Time Image Processing，2020，17（6）：2153-2162.
[38] 武慧敏.视频中的非特定异常事件时空位置检测[D].成都：电子科技大学，2018.
WU H M.A research of unspecified anomaly detection and localization in surveillance videos[D].Chengdu：University of Electronic Science and Technology of China，2018.
[39] GANOKRATANAA T，ARAMVITH S，SEBE N.Unsupervised anomaly detection and localization based on deep spatiotemporal translation network[J].IEEE Access，2020，8：50312-50329.
[40] ALAFIF T，ALZAHRANI B，CAO Y，et al.Generative adversarial network based abnormal behavior detection in massive crowd videos：A Hajj case study[J].Journal of Ambient Intelligence and Humanized Computing，2021，302：1-12.
[41] GANOKRATANAA T，ARAMVITH S，SEBE N.Anomaly event detection using generative adversarial network for surveillance videos[C]//Proceedings of 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference，Lanzhou，Nov 18?21，2019.Piscataway：IEEE，2019：1395-1399.
[42] FAN S，CHEN Y.Anomaly detection based on video prediction and latent space constraints[C]//Proceedings of International Conference on Intelligent Computing，Shenzhen，Aug 12-15，2021.Berlin：Springer，2021：535-546.
[43] FERNANDO T，DENMAN S，SRIDHARAN S，et al.Tracking by prediction：A deep generative model for mutli-person localisation and tracking[C]//Proceedings of 2018 IEEE Winter Conference on Applications of Computer Vision（WACV），Lake Tahoe，March 2-15，2018.Piscataway：IEEE，2018：1122-1132.
[44] FERNANDO T，DENMAN S，SRIDHARAN S，et al.GD-GAN：Generative adversarial networks for tra-jectory prediction and group detection in crowds[C]//Proceedings of Asian Conference on Computer Vision，Perth，Dec 2-6，2018.Berlin：Springer，2018：314-330.
[45] SONG B，SHENG R.Crowd counting and abnormal behavior detection via multiscale GAN network combined with deep optical flow[J].Mathematical Problems in Engineering，2020（8）：1-11.
[46] 刘文杰.人群计数和人群异常行为检测方法研究[D].西安：西安电子科技大学，2019.
LIU W J.Research on crowd counting and crowd abnormal behavior detection methods[D].Xi’an：Xidian University，2019.
[47] FENG X，SONG D，CHEN Y，et al.Convolutional transformer based dual discriminator generative adversarial networks for video anomaly detection[C]//Proceedings of the 29th ACM International Conference on Multimedia，Chengdu，Oct 20?24，2021.New York：ACM，2021：5546-5554.
[48] AQEEL M，KHAN K B，AZAM M A，et al.Detection of anomaly in videos using convolutional autoencoder and generative adversarial network model[C]//Proceedings of 2020 IEEE 23rd International Multitopic Conference，Bahawalpur，Nov 5-7，2020.Piscataway：IEEE，2020：1-6.
[49] WANG D L，WANG S Y.Abnormal event detection algorithm based on dual attention future frame prediction and gap fusion discrimination[J].Journal of Electronic Imaging，2021，30（2）：023009.
[50] FAN Y X，WEN G J，XIAO F，et al.Detecting anomalies in videos using perception generative adversarial network[J/OL].Circuits，Systems，and Signal Processing[2021-09-15].https：//doi.org/10.1007/s00034-021-01820-8.
[51] 杨辉，权冀川，梁新宇，等.基于弱监督学习的目标检测研究进展[J].计算机工程与应用，2021，57（16）：40-49.
YANG H，QUAN J C，LIANG X Y，et al.Research progress of object detection based on weakly supervised learning[J].Computer Engineering and Applications，2021，57（16）：40-49.
[52] MAHADEVAN V，LI W，BHALODIA V，et al.Anomaly detection in crowded scenes[C]//Proceedings of 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition，Beijing，Oct 24-28，2010.Piscataway：IEEE，2010：1975-1981.
[53] MEHRAN R，OYAMA A，SHAH M.Abnormal crowd behavior detection using social force model[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition，Miami，June 20-25，2009.Piscataway：IEEE，2009：935-942.
[54] ADAM A，RIVLIN E，SHIMSHONI I，et al.Robust real-time unusual event detection using multiple fixed-location monitors[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2008，30（3）：555-560.
[55] LU C，SHI J，JIA J.Abnormal event detection at 150?fps in matlab[C]//Proceedings of the IEEE International Conference on Computer Vision，Sydney，Dec 1-8，2013.Piscataway：IEEE，2013：2720-2727.
[56] SULTANI W，CHEN C，SHAH M.Real-world anomaly detection in surveillance videos[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Salt Lake City，June 18-23，2018.Piscataway：IEEE，2018：6479-6488.
[57] LUO W，LIU W，GAO S.A revisit of sparse coding based anomaly detection in stacked RNN framework[C]//Proceedings of the IEEE International Conference on Computer Vision，Venice，Oct 22-29，2017.Piscataway：IEEE，2017：341-349.
[58] LEYVA R，SANCHEZ V，LI C T.The LV dataset：A realistic surveillance video dataset for abnormal event detection[C]//Proceeding of 2017 5th International Workshop on Biometrics and Forensics（IWBF），Coventry，April 4-5，2017.Piscataway：IEEE，2017：1-6.
[59] 杨灿.一种结合GAN和伪标签的深度半监督模型研究[J].中国科技信息，2020（17）：79-83.
YANG C.Research on a deep semi-supervised model combining GAN and pseudo-labels[J].China Science and Technology Information，2020（17）：79-83.
[60] 王冬丽，廖春江，牟金震，等.基于特征融合的SSD视觉小目标检测[J].计算机工程与应用，2020，56（16）：31-36.
WANG D L，LIAO C J，MOU J Z，et al.SSD visual small target detection based on feature fusion[J].Computer Engineering and Applications，2020，56（16）：31-36.
[61] MI Y，ZHANG X，LI Z，et al.Dual-branch network with a subtle motion detector for microaction recognition in videos[J].IEEE Transactions on Image Processing，2020，29：6194-6208.