婴儿情感识别综述

doi:10.3778/j.issn.1002-8331.2403-0174

摘要/Abstract

摘要： 婴儿情感在塑造婴儿的认知和自我意识中扮演着关键角色，对婴儿成长具有至关重要的影响。目前婴儿情感研究领域，面临婴儿情感数据集不足、情感量化标准不统一，以及情感识别的准确性受限等问题。因此，如何有效地识别婴儿情感已成为该领域当前研究的核心难题。通过梳理当前婴儿情感识别领域的研究现状，从婴儿情感数据集、婴儿情感表示模型以及婴儿情感识别方法等方面对婴儿情感识别研究进行了分析与总结。列举并介绍了当前可用的公开数据集。总结并分析了常见婴儿情感模型，以及不同模态下的婴儿情感识别方法。概括该领域当前存在的问题以及探讨未来的研究方向，旨在为进一步的研究提供方向。

关键词: 婴儿情感, 情感数据集, 婴儿情感表示模型, 婴儿情感识别

Abstract: Infant emotions significantly influence their cognitive development and self-awareness, playing a crucial role in their growth. The field of infant emotion research currently confronts challenges such as insufficient emotion datasets, non-uniform quantification standards, and limited accuracy in emotion recognition. Therefore, effectively recognizing infant emotions has become a central issue in this field. This paper reviews and summarizes current research on infant emotion recognition, focusing on infant emotion datasets, representation models, and classification methods across various modalities. Initially, it enumerates and introduces available public datasets. Subsequently, it analyzes common models for representing infant emotions and methods for classifying them. Finally, it outlines existing challenges and discusses future research directions, aiming to guide further studies.

Key words: infant emotions, emotion datasets, infant emotion representation models, infant emotion recognition

才让卓玛, 刘鹏飞, 马光祥. 婴儿情感识别综述[J]. 计算机工程与应用, 2024, 60(19): 32-45.

Cairang Zhuoma, LIU Pengfei, MA Guangxiang. Survey of Infant Emotion Recognition[J]. Computer Engineering and Applications, 2024, 60(19): 32-45.

参考文献

[1] 王卫平, 孙锟, 常立文, 等. 儿科学[M]. 9版. 北京：人民卫生出版社, 2018.
WANG W P, SUN K, CHANG L W, et al. Pediatrics[M]. 9th ed. Beijing: People’s Medical Publishing House, 2018.
[2] PETITTO L A, KATERELOS M, LEVY B G, et al. Bilingual signed and spoken language acquisition from birth: implications for the mechanisms underlying early bilingual language acquisition[J]. Journal of Child Language, 2001, 28(2): 453-496.
[3] MALATESTA C Z, WILSON A. Emotion cognition interaction in personality development: a discrete emotions, functionalist analysis[J]. British Journal of Social Psychology, 1988, 27(1): 91-112.
[4] DARWIN C. A biographical sketch of an infant[J]. Mind, 1877, 2(7): 285-294.
[5] BRIDGES K M B. Emotional development in early infancy[J]. Child Development, 1932: 324-341.
[6] HATFIELD L A, ELY E A. Measurement of acute pain in infants: a review of behavioral and physiological variables[J]. Biological Research for Nursing, 2015, 17(1): 100-111.
[7] 徐瑄培, 张莉, 屈艺, 等. 婴儿啼哭声音的生理和病理因素分析[J]. 中国儿童保健杂志, 2018, 26(8): 869-872.
XU X P, ZHANG L, QU Y, et al. Research progress on physiological and pathological factors of infant cry[J]. Chinese Journal of Child Health Care, 2018, 26(8): 869-872.
[8] ZAMZMI G, KASTURI R, GOLDGOF D, et al. A review of automated pain assessment in infants: features, classification tasks, and databases[J]. IEEE Reviews in Biomedical Engineering, 2017, 11: 77-96.
[9] JI C, MUDIYANSELAGE T B, GAO Y, et al. A review of infant cry analysis and classification[J]. EURASIP Journal on Audio, Speech, and Music Processing, 2021, 2021(1): 8.
[10] JEYARAMAN S, MUTHUSAMY H, KHAIRUNIZAM W, et al. A review: survey on automatic infant cry analysis and classification[J]. Health and Technology, 2018, 8: 391-404.
[11] FILIPPINI C, MERLA A. Systematic review of affective computing techniques for infant robot interaction[J]. International Journal of Social Robotics, 2023, 15(3): 393-409.
[12] SCASSELLATI B, BRAWER J, TSUI K, et al. Teaching language to deaf infants with a robot and a virtual human[C]//Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 2018: 1-13.
[13] KOKKONI E, MAVROUDI E, ZEHFROOSH A, et al. GEARing smart environments for pediatric motor rehabilitation[J]. Journal of Neuroengineering and Rehabilitation, 2020, 17: 1-15.
[14] LI J, HASEGAWA-JOHNSON M, MCELWAIN N L. Analysis of acoustic and voice quality features for the classification of infant and mother vocalizations[J]. Speech Communication, 2021, 133: 41-61.
[15] 程刚, 张大均, 关雨生, 等. 婴儿面孔表情标准化图片系统的初步建立[J]. 中国心理卫生杂志, 2015, 29(6): 406-412.
CHENG G, ZHANG D J, GUAN Y S, et al. Preliminary establishment of the standardized Chinese infant facial expression of emotion[J]. Chinese Mental Health Journal, 2015, 29(6): 406-412.
[16] NATHANI S, ERTMER D J, STARK R E. Assessing vocal development in infants and toddlers[J]. Clinical Linguistics & Phonetics, 2006, 20(5): 351-369.
[17] BRAHNAM S, CHUANG C F, SEXTON R S, et al. Machine assessment of neonatal facial expressions of acute pain[J]. Decision Support Systems, 2007, 43(4): 1242-1254.
[18] BRAHNAM S, NANNI L, MCMURTREY S, et al. Neonatal pain detection in videos using the iCOPEvid dataset and an ensemble of descriptors extracted from Gaussian of local descriptors[J]. Applied Computing and Informatics, 2023, 19(1/2): 122-143.
[19] YAN J, LU G, LI X, et al. FENP: a database of neonatal facial expression for pain analysis[J]. IEEE Transactions on Affective Computing, 2020, 14(1): 245-254.
[20] GRUNAU R E, OBERLANDER T, HOLSTI L, et al. Bedside application of the neonatal facial coding system in pain assessment of premature infants[J]. Pain, 1998, 76(3): 277-286.
[21] WEBB R, AYERS S, ENDRESS A. The city infant faces database: a validated set of infant facial expressions[J]. Behavior Research Methods, 2018, 50: 151-159.
[22] MAACK J K, BOHNE A, NORDAHL D, et al. The Tromso infant faces database (TIF): development, validation and application to assess parenting experience on clarity and intensity ratings[J]. Frontiers in Psychology, 2017, 8: 409.
[23] REYES-GALAVIZ O F, CANO-ORTIZ S D, REYES-GARCÍA C A. Evolutionary-neural system to classify infant cry units for pathologies identification in recently born babies[C]//Proceedings of the 2008 Seventh Mexican International Conference on Artificial Intelligence, 2008: 330-335.
[24] PLUTCHIK R. The psychology and biology of emotion[M].[S.l.]: HarperCollins College Publishers, 1994.
[25] ACKERMAN B P, ABE J A A, IZARD C E. Differential emotions theory and emotional development: mindful of modularity[M]//What develops in emotional development. New York: Springer, 1998: 85-106.
[26] PETITTO L A, HOLOWKA S, SERGIO L E, et al. Language rhythms in baby hand movements[J]. Nature, 2001, 413(6851): 35-36.
[27] IZARD C E. The maximally discriminitive facial movements coding system, MAX[Z]. Newark :University of Delaware, 1979.
[28] IZARD C E, DOUGHERTY L M, HEMBREE E A. A system for identifying affect expressions by holistic judgments (AFFEX)[Z]. Newark: University of Delaware. Instructional Resources Center, 1983.
[29] DONDI M. Pre-Conference Workshop_Baby FACS: facial action coding system for infants and young children[C]//Proceedings of the International Congress of Infant Studies, 2022.
[30] DUNSTAN P. Calm the crying: the secret baby language that reveals the hidden meaning behind an infant’s cry[M]. [S.l.]: Penguin, 2012.
[31] ADDYMAN C, ADDYMAN I. The science of baby laughter[J]. Comedy Studies, 2013, 4(2): 143-153.
[32] 程锐, 杨洋, 史源, 等. 新生儿疼痛评估与镇痛管理专家共识（2020版）[J]. 中国当代儿科杂志, 2020, 22(9): 923-930.
CHENG R, YANG Y, SHI Y, et al. Expert consensus on neonatal pain assessment and analgesia management (2020 edition)[J]. Chinese Journal of Contemporary Pediatrics, 2020, 22(9): 923-930.
[33] 乔建中. 情绪研究: 理论与方法[M]. 南京：南京师范大学出版社, 2003.
QIAO J Z. Emotion research: theory and methods[M]. Nanjing: Nanjing Normal University Press, 2003.
[34] EKMAN P, ROSENBERG E L.What the face reveals: basic and applied studies of spontaneous expression using the facial action coding system (FACS)[M]. New York: Oxford University Press, 1997.
[35] HAMMAL Z, CHU W S, COHN J F, et al. Automatic action unit detection in infants using convolutional neural network[C]//Proceedings of the 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), 2017: 216-221.
[36] EKMAN P, DAVIDSON R J, FRIESEN W V. The Duchenne smile: emotional expression and brain physiology: II[J]. Journal of Personality and Social Psychology, 1990, 58(2): 342.
[37] ONAL ERTUGRUL I, AHN Y A, BILALPUR M, et al. Infant AFAR: automated facial action recognition in infants[J]. Behavior Research Methods, 2023, 55(3): 1024-1035.
[38] OSTER H. Automated facial analysis of infant pain expressions: progress and future directions[J]. The Lancet Digital Health, 2021, 3(10): 613-614.
[39] MAROULIS A. Baby FaceReader AU classification for infant facial expression configurations[J]. Measuring Behavior, 2018.
[40] DEWI S P, PRASASTI A L, IRAWAN B. The study of baby crying analysis using MFCC and LFCC in different classification methods[C]//Proceedings of the 2019 IEEE International Conference on Signals and Systems (ICSigSys), 2019: 18-23.
[41] BRAHNAM S, CHUANG C F, SHIH F Y, et al. Machine recognition and representation of neonatal facial displays of acute pain[J]. Artificial Intelligence in Medicine, 2006, 36(3): 211-222.
[42] 卢官明, 李晓南, 李海波. 新生儿疼痛面部表情识别方法的研究[J]. 光学学报, 2008(11): 2109-2114.
LU G M, LI X N, LI H B. Research on recognition for facial expression of pain in neonates[J]. Acta Optica Sinica, 2008(11): 2109-2114.
[43] TIPPING M E. Sparse Bayesian learning and the relevance vector machine[J]. Journal of Machine Learning Research, 2001, 1: 211-244.
[44] GHOLAMI B, HADDAD W M, TANNENBAUM A R. Relevance vector machine learning for neonate pain intensity assessment using digital imaging[J]. IEEE Transactions on Biomedical Engineering, 2010, 57(6): 1457-1466.
[45] MANSOR M N, SYAM S H F, REJAB M N. AR model for infant pain anxiety recognition using fuzzy k-NN[C]//Proceedings of the 2012 International Symposium on Instrumentation & Measurement, Sensor Network and Automation (IMSNA), 2012: 374-376.
[46] ZAMZAMI G, RUIZ G, GOLDGOF D, et al. Pain assessment in infants: towards spotting pain expression based on infants’ facial strain[C]//Proceedings of the 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), 2015: 1-5.
[47] BAHDANAU D, CHO K, BENGIO Y. Neural machine translation by jointly learning to align and translate[J]. arXiv:1409.0473, 2014.
[48] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv:2010.11929, 2020.
[49] 乔欢欢. 婴儿表情识别算法研究[D]. 成都: 电子科技大学, 2019.
QIAO H H. A research of infant expression recognition [D]. Chengdu: University of Electronic Science and Technology of China, 2019.
[50] ZHANG L, XU C, LI S. Facial expression recognition of infants based on multi-stream CNN fusion network[C]//Proceedings of the 2020 IEEE 5th International Conference on Signal and Image Processing (ICSIP), 2020: 37-41.
[51] 邓源, 施一萍, 江悦莹, 等. 基于MobileNetV2与LBP特征融合的婴幼儿表情识别算法[J]. 电子科技, 2022, 35(8): 47-52.
DENG Y, SHI Y P, JIANG Y Y, et al. Infant expression recognition algorithm based on MobileNetV2 and LBP feature fusion[J]. Electronic Science and Technology, 2022, 35(8): 47-52.
[52] LIN Q, HE R, JIANG P. Feature guided CNN for baby’s facial expression recognition[J]. Complexity, 2020, 2020: 1-10.
[53] HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 7132-7141.
[54] YAN S, ZHENG W, TANG C, et al. ARL-IL CNN for automatic facial expression recognition of infants under 24 months of age[J]. Journal of Physics: Conference Series, 2020, 1518(1): 012027.
[55] CAI J, MENG Z, KHAN A S, et al. Island loss for learning discriminative features in facial expression recognition[C]//Proceedings of the 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), 2018: 302-309.
[56] 耿磊, 齐婷婷, 张芳, 等. 基于MIFNet的婴儿面部表情识别 [J]. 计算机系统应用, 2023, 32(8): 42-53.
GENG L, QI T T, ZHANG F, et al. Facial expression recognition of infants based on MIFNet [J]. Computer Systems & Applications, 2023, 32(8): 42-53.
[57] LIU Z, LIN Y, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 10012-10022.
[58] LI X, WANG W, HU X, et al. Selective kernel networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 510-519.
[59] LU G, CHEN H, WEI J, et al. Video-based neonatal pain expression recognition with cross-stream attention[J]. Multimedia Tools and Applications, 2024, 83(2): 4667-4690.
[60] LI C, LI X, LI T. Infant facial expression recognition based on parameter-free attention module[C]//Proceedings of the 2022 International Conference on Artificial Intelligence and Computer Information Technology (AICIT), 2022: 1-4.
[61] WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 3-19.
[62] LI C, POURTAHERIAN A, VAN ONZENOORT L, et al. Infant facial expression analysis: towards a real-time video monitoring system using R-CNN and HMM[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 25(5): 1429-1440.
[63] NING M, ERTUGRUL I O, MESSINGER D S, et al. Automated emotional valence estimation in infants with stochastic and strided temporal sampling[C]//Proceedings of the 2023 11th International Conference on Affective Computing and Intelligent Interaction (ACII), 2023: 1-8.
[64] TONG Z, SONG Y, WANG J, et al. VideoMAE: masked autoencoders are data-efficient learners for self-supervised video pre-training[C]//Advances in Neural Information Processing Systems, 2022, 35: 10078-10093.
[65] BANO S, RAVIKUMAR K M. Decoding baby talk: a novel approach for normal infant cry signal classification[C]//Proceedings of the 2015 International Conference on Soft-Computing and Networks Security (ICSNS), 2015: 1-4.
[66] FELIPE G Z, AGUIAR R L, COSTA Y M G, et al. Identification of infants’ cry motivation using spectrograms[C]//Proceedings of the 2019 International Conference on Systems, Signals and Image Processing (IWSSIP), 2019: 181-186.
[67] CHANG C Y, BHATTACHARYA S, RAJ VINCENT P M D, et al. An efficient classification of neonates cry using extreme gradient boosting-assisted grouped-support-vector network[J]. Journal of Healthcare Engineering, 2021, 2021(1): 7517313.
[68] DEHAK N, KENNY P J, DEHAK R, et al. Front-end factor analysis for speaker verification[J]. IEEE Transactions on Audio, Speech, and Language Processing, 2010, 19(4): 788-798.
[69] BĂNICĂ I A, CUCU H, BUZO A, et al. Automatic methods for infant cry classification[C]//Proceedings of the 2016 International Conference on Communications (COMM), 2016: 51-54.
[70] SHARMA K, GUPTA C, GUPTA S. Infant weeping calls decoder using statistical feature extraction and Gaussian mixture models[C]//Proceedings of the 2019 10th International Conference on Computing, Communication and Networking Technologies (ICCCNT), 2019: 1-6.
[71] PETRONI M, MALOWANY A S, JOHNSTON C C, et al. Classification of infant cry vocalizations using artificial neural networks (ANNs)[C]//Proceedings of the 1995 International Conference on Acoustics, Speech, and Signal Processing, 1995: 3475-3478.
[72] FRANTI E, ISPAS I, DASCALU M. Testing the universal baby language hypothesis-automatic infant speech recognition with CNNs[C]//Proceedings of the 2018 41st International Conference on Telecommunications and Signal Processing (TSP), 2018: 1-4.
[73] LE L, KABIR A N M H, JI C, et al. Using transfer learning, SVM, and ensemble classification to classify baby cries based on their spectrogram images[C]//Proceedings of the 2019 IEEE 16th International Conference on Mobile Ad Hoc and Sensor Systems Workshops (MASSW), 2019: 106-110.
[74] 谢湘, 张立强, 王晶. 残差网络在婴幼儿哭声识别中的应用 [J]. 电子与信息学报, 2019, 41(1): 233-239.
XIE X, ZHANG L Q, WANG J. Application of residual network to infant crying recognition[J]. Journal of Electronics & Information Technology, 2019, 41(1): 233-239.
[75] OZSEVEN T. Infant cry classification by using different deep neural network models and hand-crafted features[J]. Biomedical Signal Processing and Control, 2023, 83: 104648.
[76] MAGHFIRA T N, BASARUDDIN T, KRISNADHI A. Infant cry classification using CNN-RNN[J]. Journal of Physics: Conference Series, 2020, 1528(1): 012019.
[77] 余莉萍, 梁镇麟, 梁瑞宇. 基于改进LSTM的儿童语音情感识别模型 [J]. 计算机工程, 2020, 46(6): 40-49.
YU L P, LIANG Z L, LIANG R Y. Emotion recognition model for children speech based on improved LSTM[J]. Computer Engineering, 2020, 46(6): 40-49.
[78] 李彬. 基于图卷积神经网络和Transformer表示方法的婴儿哭声分类方法研究 [D]. 合肥：安徽大学, 2022.
LI B. Research on infant cry classification method based on graph convolutional neural network and Transformer representation[D]. Hefei: Anhui University, 2022.
[79] QIAO X, JIAO S, LI H, et al. Infant cry classification using an efficient graph structure and attention-based model[J]. Kuwait Journal of Science, 2024, 51(3): 100221.
[80] XU H, ZHANG J, DAI L. Differential time-frequency log-mel spectrogram features for vision transformer based infant cry recognition[C]//Proceedings of the Interspeech 2022, 2022: 1963-1967.
[81] MAJUMDER N, HAZARIKA D, GELBUKH A, et al. Multimodal sentiment analysis using hierarchical fusion with context modeling[J]. Knowledge-Based Systems, 2018, 161: 124-133.
[82] 程大雷, 张代玮, 陈雅茜. 多模态情感识别综述[J]. 西南民族大学学报 (自然科学版), 2022, 48(4): 440-447.
CHENG D L, ZHANG D W, CHEN Y X. Survey on multimodal emotion recognition[J]. Journal of Southwest Minzu University(Natural Science Edition), 2022, 48(4): 440-447.
[83] PAL P, IYER A N, YANTORNO R E. Emotion detection from infant facial expressions and cries[C]//Proceedings of the 2006 IEEE International Conference on Acoustics Speech and Signal Processing, 2006.
[84] XU L, KRZYZAK A, SUEN C Y. Methods of combining multiple classifiers and their applications to handwriting recognition[J]. IEEE Transactions on Systems, Man, and Cybernetics, 1992, 22(3): 418-435.
[85] FANG C Y, MA C W, CHIANG M L, et al. An infant emotion recognition system using visual and audio information[C]//Proceedings of the 2017 4th International Conference on Industrial Engineering and Applications (ICIEA), 2017: 284-291.