Review on Emotion Recognition in Crowds and Groups

doi:10.3778/j.issn.1002-8331.2405-0347

Abstract

Abstract: The recognition of group emotions has important applications in public settings, safe cities, and other scenarios. With the development of artificial intelligence, there is a growing awareness of the significant contribution of using deep learning methods to the study of group emotions. This paper systematically summarizes recent domestic and international research efforts, elaborating on the current status of research in the field of group emotion recognition. It focuses on discussing methods for feature extraction and emotion recognition research in group emotions, and provides a comprehensive comparison and evaluation of related studies using the same dataset. Based on this, it outlines the challenges and issues in this research field, summarizes commonly used optimization directions and methods for feature extraction as well as research methods. The paper helps researchers better understand the characteristics of different emotion recognition tasks and provides feasible research methods and development directions for future studies.

Key words: computer vision, deep learning, emotion recognition of groups, feature extraction, emotion model

摘要： 群体情绪识别在公众场合、平安城市等场景有着重要的应用。随着人工智能的发展，人们逐渐意识到使用深度学习的方法对群体情绪研究的重要贡献。对近年来国内外相关研究工作进行系统性归纳，详细阐述了群体情绪识别领域的研究现状。着重探讨了群体情绪的特征提取方法和情绪识别研究方法，并对使用相同数据集的相关研究进行了较为全面的比较和评价。在此基础上，梳理了该研究领域的难点和问题，总结了特征提取和研究方法常用的优化方向和手段，这有助于研究者更好地了解不同情绪识别任务的特点，并为未来的研究提供了可行的研究方法和发展方向。

关键词: 计算机视觉, 深度学习, 群体情绪识别, 特征提取, 情绪模型

JING Chao, WU Yuanyuan, XIE Tianqi, SUN Weiheng. Review on Emotion Recognition in Crowds and Groups[J]. Computer Engineering and Applications, 2024, 60(23): 49-61.

敬超, 吴媛媛, 谢天圻, 孙伟恒. 群体情绪识别研究综述[J]. 计算机工程与应用, 2024, 60(23): 49-61.

References

[1] GALLAGHER A C, CHEN T. Understanding images of groups of people[C]//Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009: 256-263.
[2] PANG X, LIANG B. Motion recognition based on Kinect for human-computer intelligent interaction[J]. Journal of Physics: Conference Series, 2019, 1187(3): 032028.
[3] ZHAO C, ZHANG G, LV L, et al. Spatial-temporal consistency based crowd emotion recognition[C]//Proceedings of the 2023 26th International Conference on Computer Supported Cooperative Work in Design, 2023: 685-690.
[4] LI X, YANG Y, LI L, et al. Using deep learning models combined with crowd emotion models to identify abnormal behaviors in crowds[J]. Journal of Physics: Conference Series, 2020, 1622(1): 012051.
[5] PATWARDHAN A. Edge based grid super-imposition for crowd emotion recognition[J]. arXiv:1610.05566, 2016.
[6] VELTMEIJER E A, GERRITSEN C, HINDRIKS K. Automatic emotion recognition for groups: a review[J]. IEEE Transactions on Affective Computing, 2023, 14(1): 89-107.
[7] WANG G, GALLAGHER A, LUO J, et al. Seeing people in social context: recognizing people and social relationships[C]//Proceedings of the 11th European Conference on Computer Vision, 2010: 169-182.
[8] SGE W, COLLINS R T, RUBACK R B. Vision-based analysis of small groups in pedestrian crowds[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(5): 1003-1016.
[9] ABHINAV D, SIMON L. Collecting large, richly annotated facial-expression databases from movies[J]. IEEE Multimedia, 2012, 19(3): 34-41.
[10] SERMANET P, EIGEN D, ZHANG X, et al. OverFeat: integrated recognition, localization and detection using convolutional networks[J]. arXiv:1312.6229, 2013.
[11] LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot multibox detector[C]//Proceedings of the 14th European Conference on Computer Vision, 2016: 21-37.
[12] ZHANG K P, ZHANG Z P, LI Z F, et al. Joint face detection and alignment using multi-task cascaded convolutional networks[J]. IEEE Signal Processing Letters, 2016, 23(10): 1499-1503.
[13] DHALL A, GOECKE R, GEDEON T. Automatic group happiness intensity analysis[J]. IEEE Transactions on Affective Computing, 2015, 6(1): 13-26.
[14] GUO X, POLANíA L F, BARNER K E. Group-level emotion recognition using deep models on image scene, faces, and skeletons[C]//Proceedings of the 19th ACM International Conference on Multimodal Interaction, 2017: 603-608.
[15] CEREKOVIC A. A deep look into group happiness prediction from images[C]//Proceedings of the 18th ACM International Conference on Multimodal Interaction, 2016: 437-444.
[16] ZHU B, GUO X, BARNER K, et al. Automatic group cohesiveness detection with multi-modal features[C]//Proceedings of the 2019 International Conference on Multimodal Interaction, 2019: 577-581.
[17] WANG K, ZENG X, YANG J, et al. Cascade attention networks for group emotion recognition with face, body and image cues[C]//Proceedings of the 20th ACM International Conference on Multimodal Interaction, 2018: 640-645.
[18] YAN J, ZHENG W, CUI Z, et al. Multi-clue fusion for emotion recognition in the wild[C]//Proceedings of the 18th ACM International Conference on Multimodal Interaction, 2016: 458-463.
[19] VONIKAKIS V, YAZICI Y, NGUYEN V D, et al. Group happiness assessment using geometric features and dataset balancing[C]//Proceedings of the 18th ACM International Conference on Multimodal Interaction, 2016: 479-486.
[20] HUANG X, DHALL A, GOECKE R, et al. Analyzing group-level emotion with global alignment kernel based approach[J]. IEEE Transactions on Affective Computing, 2019, 13(2): 713-728.
[21] GUO X, ZHU B, POLANíA L F, et al. Group-level emotion recognition using hybrid deep models based on faces, scenes, skeletons and visual attentions[C]//Proceedings of the 20th ACM International Conference on Multimodal Interaction, 2018: 635-639.
[22] MIAO Y, YANG J, ALZAHRANI B, et al. Abnormal behavior learning based on edge computing toward a crowd monitoring system[J]. IEEE Network, 2022, 36(3): 90-96.
[23] MOU W, CELIKTUTAN O, GUNES H. Group-level arousal and valence recognition in static images: face, body and context[C]//Proceedings of the 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 2015: 1-6.
[24] BOSCH A, ZISSERMAN A, MUNOZ X. Representing shape with a spatial pyramid kernel[C]//Proceedings of the 6th ACM International Conference on Image and Video Retrieval, 2007: 401-408.
[25] TAN L, ZHANG K, WANG K, et al. Group emotion recognition with individual facial emotion CNNs and global image based CNNs[C]//Proceedings of the 19th ACM International Conference on Multimodal Interaction, 2017: 549-552.
[26] LAKSHMY V, MURTHY R O V. Image based group happiness intensity analysis[C]//Proceedings of the Computational Vision and Bio Inspired Computing, 2018: 1032-1040.
[27] GHOSH S, DHALL A, SEBE N. Automatic group affect analysis in images via visual attribute and feature networks[C]//Proceedings of the 25th IEEE International Conference on Image Processing, 2018: 1967-1971.
[28] ZONG Y, ZHENG W, HUANG X, et al. Emotion recognition in the wild via sparse transductive transfer linear discriminant analysis[J]. Journal on Multimodal User Interfaces, 2016, 10: 163-172.
[29] GUPTA A, AGRAWAL D, CHAUHAN H, et al. An attention model for group-level emotion recognition[C]//Proceedings of the 20th ACM International Conference on Multimodal Interaction, 2018: 611-615.
[30] CAO Z, SIMON T, WEI S E, et al. Realtime multi-person 2D pose estimation using part affinity fields[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 7291-7299.
[31] LI J, ROY S, FENG J, et al. Happiness level prediction with sequential inputs via multiple regressions[C]//Proceedings of the 18th ACM International Conference on Multimodal Interaction, 2016: 487-493.
[32] SUN B, WEI Q, LI L, et al. LSTM for dynamic emotion and group emotion recognition in the wild[C]//Proceedings of the 18th ACM International Conference on Multimodal Interaction, 2016: 451-457.
[33] BARSADE S G, GIBSON D E. Group emotion: a view from top and bottom[J]. Research on Managing Groups and Teams, 1998: 81-102.
[34] DHALL A, GOECKE R, JOSHI J, et al. EmotiW 2016: video and group-level emotion recognition challenges[C]//Proceedings of the 18th ACM International Conference on Multimodal Interaction, 2016: 427-432.
[35] HUANG X, DHALL A, ZHAO G, et al. Riesz-based volume local binary pattern and a novel group expression model for group happiness intensity analysis[C]//Proceedings of the British Machine Vision Conference, 2015.
[36] STEIN E M, WEISS G. Introduction to fourier analysis on euclidean spaces[M]. Princeton: Princeton University Press, 1971.
[37] WANG H, KL?SER A, SCHMID C, et al. Dense trajectories and motion boundary descriptors for action recognition[J]. International Journal of Computer Vision, 2013, 103: 60-79.
[38] ZHANG X, YANG X, ZHANG W, et al. Crowd emotion evaluation based on fuzzy inference of arousal and valence[J]. Neurocomputing, 2021, 445: 194-205.
[39] VARGHESE E B, THAMPI S M. A deep learning approach to predict crowd behavior based on emotion[C]//Proceedings of the International Conference on Smart Multimedia, 2018: 296-307.
[40] RABIEE H, HAMIDREZA J, MOUSAVI H, et al. Emotion-based crowd representation for abnormality detection[J]. arXiv:1607.07646, 2016.
[41] REZAEI F, YAZDI M. Real-time crowd behavior recognition in surveillance videos based on deep learning methods[J]. Journal of Real-Time Image Processing, 2021, 18(5): 1669-1679.
[42] ZHANG Y, QIN L, JI R, et al. Exploring coherent motion patterns via structured trajectory learning for crowd mood modeling[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2016, 27(3): 635-648.
[43] LAPTEV I. On space-time interest points[J]. International Journal of Computer Vision, 2005, 64: 107-123.
[44] RATRE A. Stochastic gradient descent-whale optimization algorithm-based deep convolutional neural network to crowd emotion understanding[J]. The Computer Journal, 2020, 63(2): 267-282.
[45] GAO Q J, ZHAO Z H, XU D, et al. Review on speech emotion recognition research[J]. CAAI Transactions on Intelligent Systems, 2020, 15(1): 1-13.
[46] RUSSELL J A. A circumplex model of affect[J]. Journal of Personality and Social Psychology, 1980, 39(6): 1161.
[47] RUSSELL J A, MEHRABIAN A. Distinguishing anger and anxiety in terms of emotional response factors[J]. Journal of Consulting and Clinical Psychology, 1974, 42(1): 79-83.
[48] KHAN A S, LI Z, CAI J, et al. Group-level emotion recognition using deep models with a four-stream hybrid network[C]//Proceedings of the 20th ACM International Conference on Multimodal Interaction, 2018: 623-629.
[49] ZHANG K P, ZHANG Z P, LI Z F, et al. Joint face detection and alignment using multitask cascaded convolutional networks[J]. IEEE Signal Processing Letters, 23(10): 2016: 1499-1503.
[50] FAVARETTO R M, KNOB P, MUSSE S R, et al. Detecting personality and emotion traits in crowds from video sequences[J]. Machine Vision and Applications, 2019, 30: 999-1012.
[51] ORTONY A, CLORE G L, COLLINS A. The cognitive structure of emotions[M]. Cambridge: Cambridge University Press, 2022.
[52] TRAN D, BOURDEV L, FERGUS R, et al. Learning spatiotemporal features with 3D convolutional networks[C]//Proceedings of the IEEE International Conference on Computer Vision, 2015: 4489-4497.
[53] CHOLLET F. Xception: deep learning with depthwise separable convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 1251-1258.
[54] SZEGEDY C, IOFFE S, VANHOUCKE V, et al. Inception-V4, Inception-ResNet and the impact of residual connections on learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2017: 4278-4284.
[55] TRIPATHI G, SINGH K, VISHWAKARMA D K. Crowd emotion analysis using 2D ConvNets[C]//Proceedings of the 2020 3rd International Conference on Smart Systems and Inventive Technology, 2020: 969-974.
[56] 王昊懿. 群体情绪分类方法的研究与实现[D]. 西安: 西安电子科技大学, 2021.
WANG H Y. Research and implementation of group-level emotion classification method[D]. Xi’an: Xidian University, 2021.
[57] ABBAS A, CHALUP S K. Group emotion recognition in the wild by combining deep neural networks for facial expression classification and scene-context analysis[C]//Proceedings of the 19th ACM International Conference on Multimodal Interaction, 2017: 561-568.
[58] DHALL A, JOSHI J, SIKKA K, et al. The more the merrier: analysing the affect of a group of people in images[C]//Proceedings of the 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition, 2015: 1-8.
[59] RABIEE H, HADDADNIA J, MOUSAVI H, et al. Novel dataset for fine-grained abnormal behavior understanding in crowd[C]//Proceedings of the 2016 13th IEEE International Conference on Advanced Video and Signal Based Surveillance, 2016: 95-101.
[60] LU G, ZHANG W. Happiness intensity estimation for a group of people in images using convolutional neural networks[C]//Proceedings of the 2019 3rd International Conference on Electronic Information Technology and Computer Engineering, 2019: 1707-1710.
[61] HUANG X, DHALL A, GOECKE R, et al. Multimodal framework for analyzing the affect of a group of people[J]. IEEE Transactions on Multimedia, 2018, 20(10): 2706-2721.
[62] WEI Q, ZHAO Y, XU Q, et al. A new deep-learning framework for group emotion recognition[C]//Proceedings of the 19th ACM International Conference on Multimodal Interaction, 2017: 587-592.
[63] BAWA V S, KUMAR V. Emotional sentiment analysis for a group of people based on transfer learning with a multi-modal system[J]. Neural Computing and Applications, 2019, 31: 9061-9072.
[64] LI D, LUO R, SUN S. Group-level emotion recognition based on faces, scenes, skeletons features[C]//Proceedings of the 11th International Conference on Graphics and Image Processing, 2020: 46-51.
[65] GUO X, POLANIA L, ZHU B, et al. Graph neural networks for image understanding based on multiple cues: group emotion recognition and event recognition as use cases[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2020: 2921-2930.
[66] GUO D, WANG K, YANG J, et al. Exploring regularizations with face, body and image cues for group cohesion prediction[C]//Proceedings of the International Conference on Multimodal Interaction, 2019: 557-561.
[67] QUACH K G, LE N, DUONG C N, et al. Non-volume preserving-based fusion to group-level emotion recognition on crowd videos[J]. Pattern Recognition, 2022, 128: 108646.
[68] DHALL A, KAUR A, GOECKE R, et al. EmotiW 2018: audio-video, student engagement and group-level affect prediction[C]//Proceedings of the 20th ACM International Conference on Multimodal Interaction, 2018: 653-656.
[69] BAIG M W, BAIG M S, BASTANI V, et al. Perception of emotions from crowd dynamics[C]//Proceedings of the 2015 IEEE International Conference on Digital Signal Processing, 2015: 703-707.
[70] SáNCHEZ F L, HUPONT I, TABIK S, et al. Revisiting crowd behaviour analysis through deep learning: taxonomy, anomaly detection, crowd emotions, datasets, opportunities and prospects[J]. Information Fusion, 2020, 64: 318-335.
[71] MA J. Crowd behaviour recognition methods in video images[C]//Proceedings of the 2023 International Conference on Distributed Computing and Electrical Circuits and Electronics, 2023: 1-6.