Review on Speech Emotion Recognition Research

doi:10.3778/j.issn.1002-8331.2206-0352

Abstract

Abstract: Speech is a medium for people to convey information content and express emotional attitudes at the same time, and speech emotion recognition is an important part of human-computer interaction. Starting from the concept and historical development process of speech emotion recognition, the article reviews the research system of speech emotion recognition from six perspectives step by step. It analyzes the commonly used emotion description models, summarizes the common emotional speech databases and the characteristics of different types of databases, and studies the extraction techniques of speech emotion features. By comparing the multifaceted research of many scholars of three speech emotion recognition methods, this paper derives the posture of the expected application scenarios of speech emotion recognition methods and looks forward to the challenges and development trends of speech emotion recognition technology.

Key words: speech emotion recognition, acoustic sentiment features, emotional intelligence, sentiment speech databases, deep learning

摘要： 语音是人们传递信息内容的同时又表达情感态度的媒介，语音情感识别是人机交互的重要组成部分。由语音情感识别的概念和历史发展进程入手，从6个角度逐步展开对语音情感识别研究体系进行综述。分析常用的情感描述模型，归纳常用的情感语音数据库和不同类型数据库的特点，研究语音情感特征的提取技术。通过比对3种语音情感识别方法的众多学者的多方面研究，得出语音情感识别方法可期望应用场景的态势，展望语音情感识别技术的挑战和发展趋势。

关键词: 语音情感识别, 语音情感特征, 情感智能, 语音情感数据库, 深度学习

LUO Dehu, RAN Qiwu, YANG Chao, DOU Wang. Review on Speech Emotion Recognition Research[J]. Computer Engineering and Applications, 2022, 58(21): 40-52.

罗德虎, 冉启武, 杨超, 豆旺. 语音情感识别研究综述[J]. 计算机工程与应用, 2022, 58(21): 40-52.

References

[1] 韩文静，李海峰，阮华斌，等.语音情感识别研究进展综述[J].软件学报，2014，25（1）：37?50.
HAN W J，LI H F，RUAN H B，et al.Review on speech emotion recognition[J].Journal of Software，2014，25（1）：37-50.
[2] ZHAO S，XU M，HUANG Q，et al.Introduction to the special issue on MMAC：multimodal affective computing of large-scale multimedia data[J].IEEE MultiMedia，2021，28（2）：8-10.
[3] LI Y，TAO J，CHAO L，et al.CHEAVD：a Chinese natural emotional audio-visual database[J].Journal of Ambient Intelligence and Humanized Computing，2017，8（6）：913-924.
[4] CHOU H C，LIN W C，CHANG L C，et al.NNIME：the NTHU-NTUA Chinese interactive multimodal emotion corpus[C]//2017 Seventh International Conference on Affective Computing and Intelligent Interaction（ACII），2017：292-298.
[5] YU W，XU H，MENG F，et al.Ch-sims：a Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics，2020：3718-3727.
[6] MOHAMAD NEZAMI O，JAMSHID LOU P，KARAMI M.ShEMO：a large-scale validated database for Persian speech emotion detection[J].Language Resources and Evaluation，2019，53（1）：1-16.
[7] BARROS P，CHURAMANI N，LAKOMKIN E，et al.The omg-emotion behavior dataset[C]//2018 International Joint Conference on Neural Networks（IJCNN），2018：1-7.
[8] LOTFIAN R，BUSSO C.Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings[J].IEEE Transactions on Affective Computing，2017，10（4）：471-483.
[9] ZHALEHPOUR S，ONDER O，AKHTAR Z，et al.BAUM-1：a spontaneous audio-visual face database of affective and mental states[J].IEEE Transactions on Affective Computing，2016，8（3）：300-313.
[10] CAO H，COOPER D G，KEUTMANN M K，et al.Crema-d：crowd-sourced emotional multimodal actors dataset[J].IEEE Transactions on Affective Computing，2014，5（4）：377-390.
[11] ZADEH A A B，LIANG P P，PORIA S，et al.Multimodal language analysis in the wild：cmu-mosei dataset and interpretable dynamic fusion graph[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics（Volume 1：Long Papers），2018：2236-2246.
[12] PARADA-CABALEIRO E，COSTANTINI G，BATLINER A，et al.DEMoS：an Italian emotional speech corpus[J].Language Resources and Evaluation，2020，54（2）：341-383.
[13] GERCZUK M，AMIRIPARIAN S，OTTL S，et al.EmoNet：a transfer learning framework for multi-corpus speech emotion recognition[J].arXiv：2103.08310.2021.
[14] NICOLAOU M A，PAVLOVIC V，PANTIC M.Dynamic probabilistic CCA for analysis of affective behavior and fusion of continuous annotations[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2014，36（7）：1299-1311.
[15] TRUONG K P，VAN LEEUWEN D A，DE JONG F M G.Speech-based recognition of self-reported and observed emotion in a dimensional space[J].Speech Communication，2012，54（9）：1049-1063.
[16] STAPPEN L，SCHUMANN L，SERTOLLI B，et al.MuSe-Toolbox：the multimodal sentiment analysis continuous annotation fusion and discrete class transformation toolbox[C]//Proceedings of the 2nd Conference on Multimodal Sentiment Analysis Challenge，2021：75-82.
[17] DENG J，XU X，ZHANG Z，et al.Semisupervised autoencoders for speech emotion recognition[J].IEEE/ACM Transactions on Audio，Speech，and Language Processing，2017，26（1）：31-43.
[18] SONG P，ZHENG W，YU Y，et al.Speech emotion recognition based on robust discriminative sparse regression[J].IEEE Transactions on Cognitive and Developmental Systems，2020，13（2）：343-353.
[19] TRIGEORGIS G，RINGEVAL F，BRUECKNER R，et al.Adieu features? end-to-end speech emotion recognition using a deep convolutional recurrent network[C]//2016 IEEE International Conference on Acoustics，Speech and Signal Processing（ICASSP），2016：5200-5204.
[20] LUENGO I，NAVAS E，HERNáEZ I，et al.Automatic emotion recognition using prosodic parameters[C]//Ninth European Conference on Speech Communication and Technology，2005.
[21] ILIOU T，ANAGNOSTOPOULOS C N.Statistical evaluation of speech features for emotion recognition[C]//2009 Fourth International Conference on Digital Telecommunications，2009：121-126.
[22] LIU G K.Evaluating gammatone frequency cepstral coefficients with neural networks for emotion recognition from speech[J].arXiv：1806.09010.2018.
[23] JIANG L，TAN P，YANG J，et al.Speech emotion recognition using emotion perception spectral feature[J].Concurrency and Computation：Practice and Experience，2021，33（11）：e5427.
[24] TAO H，LIANG R，ZHA C，et al.Spectral features based on local Hu moments of Gabor spectrograms for speech emotion recognition[J].IEICE Transactions on Information and Systems，2016，99（8）：2186-2189.
[25] PENG Z，DANG J，UNOKI M，et al.Multi-resolution modulation-filtered cochleagram feature for LSTM-based dimensional emotion recognition from speech[J].Neural Networks，2021，140：261-273.
[26] GUIDI A，GENTILI C，SCILINGO E P，et al.Analysis of speech features and personality traits[J].Biomedical Signal Processing and Control，2019，51：1-7.
[27] BORCHERT M，DUSTERHOFT A.Emotions in speech-experiments with prosody and quality features in speech for use in categorical and dimensional emotion recognition environments[C]//2005 International Conference on Natural Language Processing and Knowledge Engineering，2005：147-151.
[28] ZHANG S.Emotion recognition in Chinese natural speech by combining prosody and voice quality features[C]//International Symposium on Neural Networks.Berlin，Heidelberg：Springer，2008：457-464.
[29] JACOB A.Speech emotion recognition based on minimal voice quality features[C]//2016 International Conference on Communication and Signal Processing（ICCSP），2016：886-890.
[30] LI X，TAO J，JOHNSON M T，et al.Stress and emotion classification using jitter and shimmer features[C]//2007 IEEE International Conference on Acoustics，Speech and Signal Processing（ICASSP’07），2007：10811084.
[31] WANG K，AN N，LI B N，et al.Speech emotion recognition using Fourier parameters[J].IEEE Transactions on Affective Computing，2015，6（1）：69-75.
[32] RAO K S，KOOLAGUDI S G，VEMPADA R R.Emotion recognition from speech using global and local prosodic features[J].International Journal of Speech Technology，2013，16（2）：143-160.
[33] SCHULLER B，RIGOLL G，LANG M.Hidden Markov model-based speech emotion recognition[C]//2003 IEEE International Conference on Acoustics，Speech，and Signal Processing，2003.
[34] SHAMI M T，KAMEL M S.Segment-based approach to the recognition of emotions in speech[C]//2005 IEEE International Conference on Multimedia and Expo，2005：4.
[35] ZHAO H，GAO Y，XIAO Y.Upgraded attention-based local feature learning block for speech emotion recognition[C]//Pacific-Asia Conference on Knowledge Discovery and Data Mining.Cham：Springer，2021：118-130.
[36] BUSSO C，LEE S，NARAYANAN S.Analysis of emotionally salient aspects of fundamental frequency for emotion detection[J].IEEE Transactions on Audio，Speech，and Language Processing，2009，17（4）：582-596.
[37] NWE T L，FOO S W，DE SILVA L C.Speech emotion recognition using hidden Markov models[J].Speech Communication，2003，41（4）：603-623.
[38] WU S，FALK T H，CHAN W Y.Automatic speech emotion recognition using modulation spectral features[J].Speech Communication，2011，53（5）：768-785.
[39] 唐闺臣，冯月芹，梁瑞宇，等.面向语音情感识别的语谱特征提取算法研究[J].计算机工程与应用，2016，52（21）：152-156.
TANG G C，FENG Y Q，LIANG R Y，et al.Research on algorithm of spectral feature extraction for speech emotion recognition[J].Computer Engineering and Applications，2016，52（21）：152-156.
[40] HAN K，YU D，TASHEV I.Speech emotion recognition using deep neural network and extreme learning machine[C]//Proceedings of Interspeech 2014，2014：223-227.
[41] LUO D，ZOU Y，HUANG D.Investigation on joint representation learning for robust feature extraction in speech emotion recognition[C]//Interspeech，2018：152-156.
[42] WU X，CAO Y，LU H，et al.Speech emotion recognition using sequential capsule networks[J].IEEE/ACM Transactions on Audio，Speech，and Language Processing，2021，29：3280-3291.
[43] EYBEN F，WENINGER F，GROSS F，et al.Recent developments in opensmile，the munich open-source multimedia feature extractor[C]//Proceedings of the 21st ACM International Conference on Multimedia，2013：835-838.
[44] JAHANGIR R，TEH Y W，HANIF F，et al.Deep learning approaches for speech emotion recognition：state of the art and research challenges[J].Multimedia Tools and Applications，2021，80：23813.
[45] BANDELA S R，KUMAR T K.Stressed speech emotion recognition using feature fusion of teager energy operator and MFCC[C]//2017 8th International Conference on Computing，Communication and Networking Technologies（ICCCNT），2017：1-5.
[46] RAO K S，KOOLAGUDI S G.Robust emotion recognition using sentence，word and syllable level prosodic features[M]//Robust emotion recognition using spectral and prosodic features.New York，NY：Springer，2013：47-69.
[47] ARA?O K A，GLOOR P，ORSENIGO C，et al.When old meets new：emotion recognition from speech signals[J].Cognitive Computation，2021，13（3）：771-783.
[48] 王怡，王黎明，柴玉梅.融合多特征的语音情感识别方法[J].小型微型计算机系统，2022，43（6）：1232-1239.
WANG Y，WANG L M，CHAI Y M.Speech emotion recognition method fusion with multi-feature[J].Journal of Chinese Computer Systems，2022，43（6）：1232-1239.
[49] WANG C，REN Y，ZHANG N，et al.Speech emotion recognition based on multi-feature and multi-lingual fusion[J].Multimedia Tools and Applications，2021：1-11.
[50] SUN L，CHEN J，XIE K，et al.Deep and shallow features fusion based on deep convolutional neural network for speech emotion recognition[J].International Journal of Speech Technology，2018，21（4）：931-940.
[51] YU C，TIAN Q，CHENG F，et al.Speech emotion recognition using support vector machines[C]//International Conference on Computer Science and Information Engineering.Berlin，Heidelberg：Springer，2011：215-220.
[52] SEEHAPOCH T，WONGTHANAVASU S.Speech emotion recognition using support vector machines[C]//2013 5th International Conference on Knowledge and Smart Technology（KST），2013：86-91.
[53] AOUANI H，BEN AYED Y.Deep support vector machines for speech emotion recognition[C]//International Conference on Intelligent Systems Design and Applications.Cham：Springer，2019：406-415.
[54] LIN Y L，WEI G.Speech emotion recognition based on HMM and SVM[C]//2005 International Conference on Machine Learning and Cybernetics，2005：4898-4901.
[55] NTALAMPIRAS S.Toward language-agnostic speech emotion recognition[J].Journal of the Audio Engineering Society，2020，68（1/2）：7-13.
[56] MISHRA H K，SEKHAR C C.Variational Gaussian mixture models for speech emotion recognition[C]//2009 Seventh International Conference on Advances in Pattern Recognition，2009：183-186.
[57] XIAO Z，DELLANDREA E，DOU W，et al.Multistage classification of emotional speech motivated by a dimensional emotion model[J].Multimedia Tools and Applications.2010，46（1）：119-145.
[58] ZEHRA W，JAVED A R，JALIL Z，et al.Cross corpus multi-lingual speech emotion recognition using ensemble learning[J].Complex & Intelligent Systems，2021，7：1845-1854.
[59] ZHANG S，ZHANG S，HUANG T，et al.Speech emotion recognition using deep convolutional neural network and discriminant temporal pyramid matching[J].IEEE Transactions on Multimedia，2018，20（6）：1576-1590.
[60] NAM Y，LEE C.Cascaded convolutional neural network architecture for speech emotion recognition in noisy conditions[J].Sensors，2021，21（13）：4399.
[61] DEHAK N，KENNY P J，DEHAK R，et al.Front-end factor analysis for speaker verification[J].IEEE Transactions on Audio，Speech，and Language Processing，2010，19（4）：788-798.
[62] ZHANG T，WU J.Speech emotion recognition with i-vector feature and RNN model[C]//2015 IEEE China Summit and International Conference on Signal and Information Processing（ChinaSIP），2015：524-528.
[63] HERACLEOUS P，MOHAMMAD Y，YONEYAMA A.Deep convolutional neural networks for feature extraction in speech emotion recognition[C]//International Conference on Human-Computer Interaction.Cham：Springer，2019：117-132.
[64] WANG J，XUE M，CULHANE R，et al.Speech emotion recognition with dual-sequence LSTM architecture[C]//ICASSP 2020-2020 IEEE International Conference on Acoustics，Speech and Signal Processing（ICASSP），2020：6474-6478.
[65] 赵小蕾，毛启容，詹永照.融合功能性副语言的语音情感识别新方法[J].计算机科学与探索，2014，8（2）：186-199.
ZHAO X L，MAO Q R，ZHANG Y Z.New method of speech emotion recognition fusing functional paralanguages[J].Journal of Frontiers of Computer Science and Technology，2014，8（2）：186-199.
[66] HSU J，SU M，WU C，et al.Speech emotion recognition considering nonverbal vocalization in affective conversations[J].IEEE/ACM Transactions on Audio，Speech，and Language Processing，2021，29：1675-1686.
[67] HAN J，ZHANG Z，SCHMITT M，et al.From hard to soft：towards more human-like emotion recognition by modelling the perception uncertainty[C]//Proceedings of the 25th ACM International Conference on Multimedia，2017：890-897.
[68] ZHAO J，MAO X，CHEN L.Speech emotion recognition using deep 1D & 2D CNN LSTM networks[J].Biomedical Signal Processing and Control，2019，47：312-323.
[69] ATILA O，?ENGüR A.Attention guided 3D CNN-LSTM model for accurate speech based emotion recognition[J].Applied Acoustics，2021，182：108260.
[70] SABOUR S，FROSST N，HINTON G E.Dynamic routing between capsules[J].arXiv：1710.09829，2017.
[71] WU X，LIU S，CAO Y，et al.Speech emotion recognition using capsule networks[C]//ICASSP 2019-2019 IEEE International Conference on Acoustics，Speech and Signal Processing（ICASSP），2019：6695-6699.
[72] ZHONG X，LIU J，LI L，et al.An emotion classification algorithm based on SPT-CapsNet[J].Neural Computing and Applications，2020，32（7）：1823-1837.
[73] SHAHIN I，HINDAWI N，NASSIF A B，et al.Novel dual-channel long short-term memory compressed capsule networks for emotion recognition[J].Expert Systems with Applications，2022，188：116080.
[74] DENG J，ZHANG Z，EYBEN F，et al.Autoencoder-based unsupervised domain adaptation for speech emotion recognition[J].IEEE Signal Processing Letters，2014，21（9）：1068-1072.
[75] ESKIMEZ S E，DUAN Z，HEINZELMAN W.Unsupervised learning approach to feature analysis for automatic speech emotion recognition[C]//2018 IEEE International Conference on Acoustics，Speech and Signal Processing（ICASSP），2018：5099-5103.
[76] LATIF S，RANA R，QADIR J，et al.Variational autoencoders for learning latent representations of speech emotion：a preliminary study[J].arXiv：1712.08708，2017.
[77] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems，2017：5998-6008.
[78] WANG X，WANG M，QI W，et al.A Novel end-to-end speech emotion recognition network with stacked transformer layers[C]//ICASSP 2021-2021 IEEE International Conference on Acoustics，Speech and Signal Processing（ICASSP），2021：6289-6293.
[79] 杜静，唐曼婷，赵力.基于线性注意力和类Transformer模型的语音情感识别[J].东南大学学报（英文版），2021，37（2）：164-170.
DU J，TANG M T，ZHAO L.Transformer-like model with linear attention for speech emotion recognition[J].Journal of Southeast University（English Edition），2021，37（2）：164-170.
[80] CHEN W，XING X，XU X，et al.Key-sparse transformer with cascaded cross-attention block for multimodal speech emotion recognition[J].arXiv：2106.11532，2021.
[81] LI L，ZHAO Y，JIANG D，et al.Hybrid deep neural network—hidden markov model（DNN-HMM） based speech emotion recognition[C]//2013 Humaine Association Conference on Affective Computing and Intelligent Interaction，2013：312-317.
[82] SUN L，ZOU B，FU S，et al.Speech emotion recognition based on DNN-decision tree SVM model[J].Speech Communication，2019，115：29-37.
[83] SCHULLER B，VLASENKO B，EYBEN F，et al.Cross-corpus acoustic emotion recognition：variances and strategies[J].IEEE Transactions on Affective Computing，2010，1（2）：119-131.
[84] 宋鹏，郑文明，赵力.基于特征迁移学习方法的跨库语音情感识别[J].清华大学学报（自然科学版），2016，56（11）：1179-1183.
SONG P，ZHENG W M，ZHAO L.Cross-corpus speech emotion recognition based on a feature transfer learning method[J].Journal of Tsinghua University（Science and Technology），2016，56（11）：1179-1183.
[85] ZONG Y，ZHENG W，ZHANG T，et al.Cross-corpus speech emotion recognition based on domain-adaptive least-squares regression[J].IEEE Signal Processing Letters，2016，23（5）：585-589.
[86] LATIF S，RANA R，YOUNIS S，et al.Transfer learning for improving speech emotion classification accuracy[J].arXiv：1801.06353，2018.
[87] GANIN Y，USTINOVA E，AJAKAN H，et al.Domain-adversarial training of neural networks[J].The Journal of Machine Learning Research，2016，17（1）：1996-2030.
[88] ABDELWAHAB M，BUSSO C.Domain adversarial for acoustic emotion recognition[J].IEEE/ACM Transactions on Audio，Speech，and Language Processing，2018，26（12）：2423-2435.
[89] AHN Y，LEE S J，SHIN J W.Cross-corpus speech emotion recognition based on few-shot learning and domain adaptation[J].IEEE Signal Processing Letters，2021，28：1190-1194.
[90] SAHU S，GUPTA R，ESPY-WILSON C.Modeling feature representations for affective speech using generative adversarial networks[J].IEEE Transactions on Affective Computing，2020.
[91] LI H C，PAN T，LEE M H，et al.Make patient consultation warmer：a clinical application for speech emotion recognition[J].Applied Sciences，2021，11（11）：4782.
[92] 秦海波，白延强，吴斌，等.载人航天飞行中的情绪研究进展[J].航天医学与医学工程，2012，25（4）：302-306.
QIN H B，BAI Y Q，WU B，et al.Progress of study on emotion in manned spaceflight[J].Space Medicine & Medical Engineering，2012，25（4）：302-306.
[93] LI W，ZHANG Y，FU Y.Speech emotion recognition in e-learning system based on affective computing[C]//Third International Conference on Natural Computation（ICNC 2007），2007：809-813.
[94] LEE C M，NARAYANAN S S.Toward detecting emotions in spoken dialogs[J].IEEE Transactions on Speech and Audio Processing，2005，13（2）：293-303.
[95] NAKATSU R，NICHOLSON J，TOSA N.Emotion recognition and its application to computer agents with spontaneous interactive capabilities[C]//Proceedings of the Seventh ACM International Conference on Multimedia（Part 1），1999：343-351.
[96] HOSSAIN M S，MUHAMMAD G，SONG B，et al.Audio-visual emotion-aware cloud gaming framework[J].IEEE Transactions on Circuits and Systems for Video Technology，2015，25（12）：2105-2118.
[97] 王晶晶，黄如.基于宽深学习网络的音乐情感识别[J].华东理工大学学报（自然科学版），2022，48（3）：373-380.
WANG J J，HUANG R.Music emotion recognition based on the broad and deep learning network[J].Journal of East China University of Science and Technolog（Natural Science Edtion），2022，48（3）：373-380.
[98] MOUTTI M，SOFIA E，PANAGIOTIS K，et al.A dataset for speech emotion recognition in greek theatrical plays[J].arXiv：2203.15568，2022.
[99] TAN L，YU K，LIN L，et al.Speech emotion recognition enhanced traffic efficiency solution for autonomous vehicles in a 5G-enabled space-air-ground integrated intelligent transportation system[J].IEEE Transactions on Intelligent Transportation Systems，2021：1-13.
[100] 赵立，梁瑞宇，魏昕，等.语音信号处理[M].3版.北京：机械工业出版社，2016：232-235.
ZHAO L，LIANG R Y，WEI X，et al.Speech signal processing[M].3rd ed.Beijing：China Machine Progress，2016：232-235.
[101] RAMAKRISHNAN S，El EMARY I M M.Speech emotion recognition approaches in human computer interaction[J].Telecommunication Systems，2013，52（3）：1467-1478.
[102] CHANG Y，LARIDI S，REN Z，et al.Robust federated learning against adversarial attacks for speech emotion recognition[J].arXiv：2203.04696，2022.