Survey on Emotion Recognition in Conversation

doi:10.3778/j.issn.1002-8331.2207-0417

Abstract

Abstract: Emotion recognition in conversation（ERC） is a hot research topic in the field of emotion computing, which aims to detect the emotion category of each discourse during the dialogue. It has important research significance for dialogue understanding and dialogue generation. At the same time, it has a wide range of practical application value in many fields, such as social media analysis, recommendation system, medical treatment and human-computer interaction. With the continuous innovation and development of deep learning technology, emotion recognition in conversation has attracted more and more attention from academia and industry. At this stage, it is necessary to summarize these research results in an overview article in order to better carry out follow-up work. The research results in this field are comprehensively sorted out from the perspectives of problem definition, problem approach, research methods, and mainstream datasets, and the development of dialogue emotion recognition tasks is reviewed and analyzed. Compared with video and audio, dialogue text contains more information. Therefore, this paper focuses on combing the text dialogue emotion recognition methods, especially the methods based on deep learning. Finally, based on the current research status, this paper summarizes the open problems existing in the field of dialogue emotion recognition and the development trend in the future.

Key words: sentiment analysis, dialogue understanding, emotion recognition in conversation, deep learning

摘要： 对话情绪识别是情感计算领域的一个热门研究课题，旨在检测对话过程中每个话语的情感类别。其在对话理解和对话生成方面具有重要的研究意义，同时在社交媒体分析、推荐系统、医疗和人机交互等诸多领域具有广泛的实际应用价值。随着深度学习技术的不断创新和发展，对话情绪识别受到学术界和工业界越来越多的关注，现阶段需要综述性的文章对已有研究成果进行总结，以便更好地开展后续工作。从问题定义、问题切入方式、研究方法、主流数据集等多个角度对该领域的研究成果进行全面梳理，回顾和分析了对话情绪识别任务的发展。对话文本中含有丰富的语义信息，结合视频和音频可以进一步提升建模效果，因此，重点对文本对话情绪识别以及多模态对话情绪识别的方法进行了梳理，立足于当前研究现状，总结了现有对话情绪识别领域存在的开放问题以及未来的发展趋势。

关键词: 情感分析, 对话理解, 对话情绪识别, 深度学习

CHEN Xiaoting, LI Shi. Survey on Emotion Recognition in Conversation[J]. Computer Engineering and Applications, 2023, 59(3): 33-48.

陈晓婷, 李实. 对话情绪识别综述[J]. 计算机工程与应用, 2023, 59(3): 33-48.

References

[1] MARIETTO M，AGUIAR R，BARBOSA G，et al.Artificial intelligence markup language：a brief tutorial[J].arXiv：1307.3091，2013.
[2] GOODRICH M，SCHULTZ A.Human-robot interaction：a survey[J].Foundations and Trends in Human-Computer Interaction，2007，1（3）：203-275.
[3] MAYER J，ROBERTS R，BARSADE S.Human abilities：emotional intelligence[J].Annual Review of Psychology，2008，59（1）：507-536.
[4] RASHKIN H，SMITH E，LI M，et al.Towards empathetic open-domain conversation models：a new benchmark and dataset[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.Florence，PA：ACL，2019：5370-5381.
[5] MA Y，NGUYEN K L，XING F Z，et al.A survey on empathetic dialogue systems[J].Information Fusion，2020，64：50-70.
[6] MAJUMDER N，HONG P，PENG S，et al.MIME：mimicking emotions for empathetic response generation[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing（EMNLP）.PA：ACL，2020：8968-8979.
[7] XIE B，PARK C H.Empathetic robot with transformer-based dialogue agent[C]//2021 18th International Conference on Ubiquitous Robots（UR）.Gangneung，NJ：IEEE，2021：290-295.
[8] CHATTERJEE A，NARAHARI K N，JOSHI M，et al.SemEval-2019 Task 3：emoContext contextual emotion detection in text[C]//Proceedings of the 13th International Workshop on Semantic Evaluation.Minneapolis，PA：ACL，2019：39-48.
[9] SHEN T，JIA J，LI Y，et al.PEIA：personality and emotion integrated attentive model for music recommendation on social media platforms[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2020：206-213.
[10] ZHANG Y，LAI G，MIN Z，et al.Explicit factor models for explainable recommendation based on phrase-level sentiment analysis[C]//Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval.New York：ACM，2014：83-92.
[11] MAHATA D，FRIEDRICHS J，SHAH R R，et al.Detecting personal intake of medicine from twitter[J].IEEE Intelligent Systems，2018，33（4）：87-95.
[12] GUO C，CAO J，ZHANG X，et al.DEAN：learning dual emotion for fake news detection on social media[J].arXiv：1728.1903，2017.
[13] XIAO G，TU G，ZHENG L，et al.Multimodality sentiment analysis in social internet of things based on hierarchical attentions and CSAT-TCN with MBM network[J].IEEE Internet of Things Journal，2021，8（16）：12748-12757.
[14] COLNERIC N，DEMSAR J.Emotion recognition on twitter：comparative study and training a unison model[J].IEEE Transactions on Affective Computing，2018，11（3）：433-446.
[15] PORIA S，MAJUMDER N，MIHALCEA R，et al.Emotion recognition in conversation：research challenges，datasets，and recent advances[J].arXiv：1905.2947，2019.
[16] MUNEZERO M，SUERO MONTERO C，SUTINEN E，et al.Are they different? affect，feeling，emotion，sentiment，and opinion detection in text[J].IEEE Transactions on Affective Computing，2014，5（2）：101-111.
[17] WEI L，WEI S，JI S，et al.BiERU：bidirectional emotional recurrent unit for conversational sentiment analysis[J].arXiv：2006.2492，2020.
[18] LIAN Z，LIU B，TAO J.CTNet：conversational transformer network for emotion recognition[J].IEEE/ACM Transactions on Audio，Speech，and Language Processing，2021（29）：985-1000.
[19] MA H，WANG J，LIN H，et al.A multi-view network for real-time emotion recognition in conversations[J].Knowledge-Based Systems，2021，236：107751.
[20] PORIA S，CAMBRIA E，HAZARIKA D，et al.Context-dependent sentiment analysis in user-generated videos[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics.Vancouver，PA：ACL，2017：873-883.
[21] CHEN S，HSU C，KUO C，et al.EmotionLines：an emotion corpus of multi-party conversations[C]//Proceedings of the Eleventh International Conference on Language Resources and Evaluation，Miyazaki，2018.
[22] 吴良庆，李寿山，王建成，等.基于神经主题模型的对话情感分析[J].中文信息学报，2020，34（1）：106-112.
WU L Q，LI S S，WANG J C，et al.Dialog sentiment analysis with neural topic model[J].Journal of Chinese Information Processing，2020，34（1）：106-112.
[23] HAZARIKA D，PORIA S，ZADEH A，et al.Conversational memory network for emotion recognition in dyadic dialogue videos[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies.New Orleans，PA：ACL，2018：2122-2132.
[24] HAZARIKA D，PORIA S，MIHALCEA R，et al.ICON：interactive conversational memory network for multimodal emotion detection[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.Brussels，PA：ACL，2018：2594-2604.
[25] MAJUMDER N，PORIA S，HAZARIKA D，et al.DialogueRNN：an attentive RNN for emotion detection in conversations[J].arXiv：1405.1811，2014.
[26] GHOSAL D，MAJUMDER N，PORIA S，et al.DialogueGCN：a graph convolutional neural network for emotion recognition in conversation[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing（EMNLP-IJCNLP）.Hong Kong，PA：ACL，2019：154-164.
[27] SHEN W，CHEN J，QUAN X，et al.DialogXL：all-in-one XLNet for multi-party conversation emotion recognition[J].arXiv：2012.8695，2020.
[28] 田雨亭.基于深度学习的对话情绪识别研究[D].武汉：华中师范大学，2021.
TIAN Y T.Research on emotion recognition in conversation based on deep learning[D].Wuhan：Central China Normal University，2021.
[29] WANG Y，ZHANG J，MA J，et al.Contextualized emotion recognition in conversation as sequence tagging[C]//Proceedings of the 21st Annual Meeting of the Special Interest Group on Discourse and Dialogue.PA：ACL，2020：186-195.
[30] GAO Q，CAO B，GUAN X，et al.Emotion recognition in conversations with emotion shift detection based on multi-task learning[J].Knowledge-Based Systems，2022，248：108861.
[31] HOFMANN T.Probabilistic latent semantic analysis[C]//Proceedings of the Fifteenth Conference on Uncertainty in Artificial Intelligence.New York：ACM，1999：289-296.
[32] BLEI D，NG A，JORDAN M.Latent dirichlet allocation[J].The Journal of Machine Learning Research，2001，3：993-1022.
[33] RANA T，CHEAH Y，LETCHMUNAN S.Topic modeling in sentiment analysis：a systematic review[J].Journal of ICT Research and Applications，2016，10：76-93.
[34] XIANG B，ZHOU L.Improving twitter sentiment analysis with topic-based mixture modeling and semi-supervised training[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics.Baltimore，PA：ACL，2014：434-439.
[35] REN Y，ZHANG Y，ZHANG M，et al.Improving twitter sentiment classification using topic-enriched multi-prototype word embeddings[C]//Proceedings of the 30th Conference on Artificial Intelligence，2016：3038-3044.
[36] ZHU L，PERGOLA G，GUI L，et al.Topic-driven and knowledge-aware transformer for dialogue emotion detection[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing.PA：ACL，2021：1571-1582.
[37] JOSHI A，BHATTACHARYYA P，AHIRE S.Sentiment resources：lexicons and datasets[M]//A practical guide to sentiment analysis.[S.l.]：Springer，2017：85-106.
[38] CHEN L，LEE C，CHEN M.Exploration of social media for sentiment analysis using deep learning[J].Soft Computing，2020，24：8187-8197.
[39] PAN D，YUAN J，LI L，et al.Deep neural network-based classification model for sentiment analysis[J].arXiv：1907.2046，2019.
[40] RABEYA T，FERDOUS S，ALI H，et al.A survey on emotion detection：a lexicon based backtracking approach for detecting emotion from Bengali text[C]//20th International Conference of Computer and Information Technology（ICCIT）.Bangladesh，NJ：IEEE，2017：1-7.
[41] STRAPPARAVA C，VALITUTTI A.WordNet-affect：an affective extension of wordNet[C]//Proceedings of the Fourth International Conference on Language Resources and Evaluation（LREC），Lisbon，2004：1083-1086.
[42] MOHAMMAD S，TURNEY P.Crowdsourcing a word-emotion association lexicon[J].arXiv：1308.6297，2013.
[43] LI Z，XIE H，CHENG G，et al.Word-level emotion distribution with two schemas for short text emotion classification[J].Knowledge-Based Systems，2021，227：107163.
[44] PORIA S，GELBUKH A，CAMBRIA E，et al.Emosenticspace：a novel Framework for affective common-sense reasoning[J].Knowledge-Based Systems，2014，69：108-123.
[45] BANDHAKAVI A S，WIRATUNGA N.Lexicon based feature extraction for emotion text classification[J].Pattern Recognition Letters，2016，93：133-142.
[46] OJAMAA B，JOKINEN P A I K，MUISCHENK K.Sentiment analysis on conversational texts[C]//Proceedings of the 20th Nordic Conference of Computational Linguistics，Vilnius，Nodalida，2015：233-237.
[47] MAZZOCUT M，TRUCCOLO I，ANTONINI M，et al.Web conversations about complementary and alternative medicines and cancer：content and sentiment analysis[J].Journal of Medical Internet Research，2016，18（6）：e120.
[48] BOTHE C，MAGG S，WEBER C，et al.Dialogue-based neural learning to estimate the sentiment of a next upcoming utterance[C]//Proceedings of the 26th International Conference on Artificial Neural Networks（ICANN），Alghero，2017：477-485.
[49] SHIVHARE S N，GARG S，MISHRA A.Emotionfinder：detecting emotion from blogs and textual documents[C]//International Conference on Computing，Communication & Automation.Greater Noida，NJ：IEEE，2015：52-57.
[50] MA C，HELMUT P.Emotion estimation and reasoning based on affective textual interaction[C]//Affective Computing and Intelligent Interaction（ACII）.Berlin：Springer，2005：622-628.
[51] HERZIG J，FEIGENBLAT G，SHMUELI-SCHEUER M，et al.Classifying emotions in customer support dialogues in social media[C]//17th Annual Meeting of the Special Interest Group on Discourse and Dialogue.Los Angeles，PA：ACL，2016：64-73.
[52] JAIN V，KUMAR D S，FERNANDES S.Extraction of emotions from multilingual text using intelligent text processing and computational linguistics[J].Journal of Computational Science，2017，21：316-326.
[53] HASAN M，RUNDENSTEINER E A，AGU E O.Emotex：detecting emotions in twitter messages[C]//Bigdata Socialcom Cybersecurity Conference，2014：27-31.
[54] AMAN S，SZPAKOWICZ S.Identifying expressions of emotion in text[C]//Proceedings of the 10th International Conference on Text，Speech，and Dialogue.Berlin：Springer，2007.
[55] GHAZI D，INKPEN D，SZPAKOWICZ S.Hierarchical versus flat classification of emotions in text[C]//Proceedings of the NAACLHLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text.Stroudsburg，PA：ACL，2010：140-146.
[56] DE BRUYNE L，DE CLERCQ O E E，HOSTE V E R.LT3 at SemEval-2018 Task 1：a classifier chain to detect emotions in tweets[C]//Proceedings of The 12th International Workshop on Semantic Evaluation.New Orleans，PA：ACL，2018：123-127.
[57] SINGH L，SINGH S，AGGARWAL N.Two-stage text feature selection method for human emotion recognition[C]//Proceedings of 2nd International Conference on Communication，Computing and Networking，Lecture Notes in Networks and Systems.Singapore：Springer，2018：531-538.
[58] HERZIG J，SHMUELI-SCHEUER M，KONOPNICKI D.Emotion detection from text via ensemble classification using word embeddings[C]//Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval（ICTIR’17）.New York：ACM，2017：269-272.
[59] GUPTA U，CHATTERJEE A，SRIKANTH R，et al.A sentiment-and-semantics-based approach for emotion detection in textual conversations[J].arXiv：1707.6996，2017.
[60] HUANG C，TRABELSI A，ZA?ANE O.ANA at SemEval-2019 task 3：contextual emotion detection in conversations through hierarchical LSTMs and BERT[C]//Proceedings of the 13th International Workshop on Semantic Evaluation.Minneapolis，PA：ACL，2019：49-53.
[61] HU D，WEI L，HUAI X.DialogueCRN：contextual reasoning networks for emotion recognition in conversations[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing.PA：ACL，2021：7042-7052.
[62] ZHANG Y，TIWARI P，SONG D，et al.Learning interaction dynamics with an interactive LSTM for conversational sentiment analysis[J].Neural Networks，2021，133：40-56.
[63] JIAO W，YANG H，KING I，et al.HiGRU：hierarchical gated recurrent units for utterance-level emotion recognition[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies.Minneapolis，PA：ACL，2019：397-406.
[64] MA H，WANG J，QIAN L，et al.HAN-ReGRU：hierarchical attention network with residual gated recurrent unit for emotion recognition in conversation[J].Neural Computing and Applications，2021，33（7）：2685-2703.
[65] LI J，ZHANG M，JI D，et al.Multi-task learning with auxiliary speaker identification for conversational emotion recognition[J].arXiv：1478.2003，2014.
[66] JIANG T，XU B，ZHAO T，et al.CAN-GRU：a hierarchical model for emotion recognition in dialogue[C]//Proceedings of the 19th Chinese National Conference on Computational Linguistics.Haikou，Chinese Information Processing Society of China，2020：1101-1111.
[67] ZHANG D，CHEN X，XU S，et al.Knowledge aware emotion recognition in textual conversations via multi-task incremental transformer[C]//Proceedings of the 28th International Conference on Computational Linguistics.Barcelona，PA：ACL，2020：4429-4440.
[68] LI J，JI D，LI F，et al.HiTrans：a transformer-based context-and speaker-sensitive model for emotion detection in conversations[C]//Proceedings of the 28th International Conference on Computational Linguistics.Barcelona，International Committee on Computational Linguistics，2020：4190-4200.
[69] TU G，WEN J，LIU C，et al.Context-and sentiment-aware networks for emotion recognition in conversation[J].IEEE Transactions on Artificial Intelligence，2022，1：1.
[70] LIU H Z，LIN H Z，CHEN G.TANTP：conversational emotion recognition using tree-based attention networks with transformer pre-training[C]//Advances in Knowledge Discovery and Data Mining，2021：730-742.
[71] SCARSELLI F，GORI M，TSOI A C，et al.The graph neural network model[J].IEEE Transactions on Neural Networks，2009，20（1）：61-80.
[72] KIPF T，WELLING M.Semi-supervised classification with graph convolutional networks[J].arXiv：1609.2907，2016.
[73] SCHLICHTKRULL M，KIPF T N，BLOEM P，et al.Modeling relational data with graph convolutional networks[J].arXiv：1703.6103，2017.
[74] ISHIWATARI T，YASUDA Y，MIYAZAKI T，et al.Relation-aware graph attention networks with relational position encodings for emotion recognition in conversations[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing（EMNLP）.PA：ACL，2020：7360-7370.
[75] SHEN W，WU S，YANG Y，et al.Directed a cyclic graph network for conversational emotion recognition[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing.PA：ACL，2021：1551-1560.
[76] LIANG C，YANG C，XU J，et al.S+PAGE：a speaker and position-aware graph neural network model for emotion recognition in conversation[J].arXiv：2112.12389，2021.
[77] SUN Y，YU N，FU G.A discourse-aware graph neural network for emotion recognition in multi-party conversation[C]//Findings of the Association for Computational Linguistics.Punta Cana，PA：ACL，2021：2949-2958.
[78] ZHANG D，WU L，SUN C，et al.Modeling both context- and speaker-sensitive dependence for emotion detection in multi-speaker conversations[C]//Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence，Macao，2019：5415-5421.
[79] LIAN Z，LIU B，TAO J.DECN：dialogical emotion correction network for conversational emotion recognition[J].Neurocomputing，2021，454：6.
[80] SHENG D，WANG D，SHEN Y，et al.Summarize before aggregate：a global-to-local heterogeneous graph inference network for conversational emotion recognition[C]//Proceedings of the 28th International Conference on Computational Linguistics.Barcelona，Spain，PA：ACL，2020：4153-4163.
[81] 彭韬，杨亮，桑钟屹，等.基于异构二部图的对话情感分析[J].中文信息学报，2021，35（11）：135-142.
PENG T，YANG L，SANG Z Y，et al.Conversational sentiment analysis based on heterogeneous bipartite graphs[J].Journal of Chinese Information Processing，2021，35（11）：135-142.
[82] JIAO W，LYU M，KING I.Real-time emotion recognition via attention gated hierarchical memory network[J].arXiv：1911.9075，2019.
[83] HINTON G，KRIZHEVSKY A，WANG S.Transforming auto-encoders[C]//21st International Conference on Artificial Neural Networks，Espoo，2011：44-51.
[84] SABOUR S，FROSST N，HINTON G E.Dynamic routing between capsules[J].arXiv：1710.9829，2017.
[85] XIAO Y，LI J，ZHAO Q，et al.Emotion recognition in conversation using capsule networks and gated recurrent units[C]//International Conference on Innovative Computing，Singapore，2022：59-67.
[86] CERISARA C，JAFARITAZEHJANI S，OLUOKUN A，et al.Multi-task dialog act and sentiment recognition on Mastodon[J].arXiv：1807.5013，2018.
[87] KIM M，KIM H.Integrated neural network model for identifying speech acts，predicators，and sentiments of dialogue utterances[J].Pattern Recognition Letters，2017，101：1-5.
[88] QIN L，CHE W，LI Y，et al.DCR-Net：a deep co-interactive relation network for Joint dialog act recognition and sentiment classification[C]//The Thirty-Fourth AAAI Conference on Artificial Intelligence.New York，CA：AAAI，2020：8665-8672.
[89] QIN L，LI Z，CHE W，et al.Co-GAT：a co-interactive graph attention network for joint dialog act recognition and sentiment classification[C]//Proceedings of the 35th AAAI Conference on Artificial Intelligence.New York，CA：AAAI，2021.
[90] LI J，FEI H，JI D.Modeling local contexts for joint dialogue act recognition and sentiment classification with bi-channel dynamic convolutions[C]//Proceedings of the 28th International Conference on Computational Linguistics.Barcelona，PA：ACL，2020：616-626.
[91] XING B，TSANG I.DARER：dual-task temporal relational recurrent reasoning network for joint dialog sentiment classification and act recognition[C]//Proceedings of the 28th International Conference on Computational Linguistics.Barcelona，PA：ACL，2022：616-626.
[92] ZHANG Y，LI Q，SONG D，et al.Quantum-inspired interactive networks for conversational sentiment analysis[C]//Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence，2019：5436-5442.
[93] LI Q，GKOUMAS D，SORDONI A，et al.Quantum-inspired neural network for conversational emotion recognition[C]//Proceedings of the Thirty-Fifth AAAI Conference on Artificial Intelligence.CA：AAAI，2021：13270-13278.
[94] YAN P，LI L，ZENG D.Quantum probability-inspired graph attention network for modeling complex text interaction[J].Knowledge-Based Systems，2021，234：107557.
[95] ZHONG P，WANG D，MIAO C.Knowledge-enriched transformer for emotion detection in textual conversations[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing.Hong Kong，PA：ACL，2019：165-176.
[96] GHOSAL D，MAJUMDER N，GELBUKH A，et al.COSMIC：commonsense knowledge for emotion identification in conversations[C]//Findings of the Association for Computational Linguistics.PA：ACL，2020：2470-2481.
[97] LI J，LIN Z，FU P，et al.Past，present，and future：conversational emotion recognition through structural modeling of psychological knowledge[C]//Findings of the Association for Computational Linguistics.Punta Cana，PA：ACL，2021：1204-1214.
[98] BALTRU?AITIS T，AHUJA C，MORENCY L.Multimodal machine learning：a survey and taxonomy[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2019，41（2）：423-443.
[99] ZADEH A，CHEN M，PORIA S，et al.Tensor fusion network for multimodal sentiment analysis[J].arXiv：1707. 7250，2017.
[100] LIU Z，SHEN Y，LAKSHMINARASIMHAN V B，et al.Efficient low-rank multimodal fusion with modality-specific factors[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.Melbourne，PA：ACL，2018：2247-2256.
[101] MAI S，HU H，XING S.Divide，conquer and combine：hierarchical feature fusion network with local and global perspectives for multimodal affective computing[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.Florence，PA：ACL，2019：481-492.
[102] GU Y，YANG K，FU S，et al.Multimodal affective analysis using hierarchical attention strategy with word-level alignment[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.Melbourne，PA：ACL，2018：2225-2235.
[103] YOON S，BYUN S，DEY S，et al.Speech emotion recognition using multi-hop attention mechanism[C]//International Conference on Acoustics，Speech and Signal Processing（ICASSP）.Brighton，NJ：IEEE，2019：2822-2826.
[104] XU H，ZHANG H，HAN K，et al.Learning alignment for multimodal emotion recognition from speech[J].arXiv：1909.5645，2019.
[105] TSAI Y H，BAI S，LIANG P P，et al.Multimodal transformer for unaligned multimodal language sequences[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.Florence，PA：ACL，2019：6558-6569.
[106] WU C，LIN J，WEI W.Survey on audiovisual emotion recognition：databases，features，and data fusion strategies[J].APSIPA Transactions on Signal and Information Processing，2014，3：1-18.
[107] ROZGI? V，ANANTHAKRISHNAN S，SALEEM S，et al.Ensemble of SVM trees for multimodal emotion recognition[C]//Proceedings of The 2012 Asia Pacific Signal and Information Processing Association Annual Summit and Conference.Hollywood，NJ：IEEE，2012：1-4.
[108] PéREZ ROSAS V，MIHALCEA R，MORENCY L.Multimodal sentiment analysis of spanish online videos[J].IEEE Intelligent Systems，2013，28（3）：38-45.
[109] JIN Q，LI C，CHEN S，et al.Speech emotion recognition with acoustic and lexical features[C]//2015 IEEE International Conference on Acoustics，Speech and Signal Processing（ICASSP）.South Brisbane，NJ：IEEE，2015：4749-4753.
[110] SCHULLER B，VALSTAR M，COWIE R，et al.AVEC 2012：the continuous audio/visual emotion challenge[C]//Proceedings of the ACM International Conference on Multimodal Interaction.California：ACM，2012.
[111] HUANG J，LI Y，TAO J，et al.Continuous multimodal emotion prediction based on long short term memory recurrent neural network[C]//Proceedings of the 7th Annual Workshop on Audio/Visual Emotion Challenge.New York：ACM，2017：11-18.
[112] KESSOUS L，CASTELLANO G，CARIDAKIS G.Multimodal emotion recognition in speech-based interaction using facial expression，body gesture and acoustic analysis[J].Journal on Multimodal User Interfaces，2009，3（1）：33-48.
[113] HUANG Y，YANG J，LIAO P，et al.Fusion of facial expressions and EEG for multimodal emotion recognition[J].Computational Intelligence and Neuroscience，2017（2）：1-8.
[114] MORVANT E，HABRARD A，AYACHE S.Majority vote of diverse classifiers for late fusion[J].arXiv：1404. 7796，2014.
[115] SHUTOVA E，KIELA D，MAILLARD J.Black holes and white rabbits：metaphor identification with visual features[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies.San Diego，PA：ACL，2016：160-170.
[116] GLODEK M，TSCHECHNE S，LAYHER G，et al.Multiple classifier systems for the classification of audio-visual emotional states[J].Lecture Notes in Computer Science，2011，6975：359-368.
[117] CHEN S，JIN Q.Multi-modal conditional attention fusion for dimensional emotion prediction[C]//Proceedings of the 24th ACM International Conference on Multimedia.New York：ACM，2016：571-575.
[118] PORIA S，CAMBRIA E，HAZARIKA D，et al.Multi-level multiple attentions for contextual multimodal sentiment analysis[C]//2017 IEEE International Conference on Data Mining（ICDM）.New Orleans，NJ：IEEE，2017：1033-1038.
[119] CHOI J，LEE J.EmbraceNet：a robust deep learning architecture for multimodal classification[J].Information Fusion，2019，51：259-270.
[120] XIE B，SIDULOVA M，PARK C H.Robust multimodal emotion recognition from conversation with transformer-based crossmodality fusion[J].Sensors，2021，21（14）：4913.
[121] LIAN Z，LIU B，TAO J.CTNet：conversational transformer network for emotion recognition[J].IEEE/ACM Transactions on Audio，Speech，and Language Processing，2021，29：985-1000.
[122] AGARWAL H，BANSAL K，JOSHI A，et al.Shapes of emotions：multimodal emotion recognition in conversations via emotion shifts[J].arXiv：1938.2112，2019.
[123] REN M，HUANG X，SHI X，et al.Interactive multimodal attention network for emotion recognition in conversation[J].IEEE Signal Processing Letters，2021，28：1046-1050.
[124] HU J，LIU Y，ZHAO J，et al.MMGCN：multimodal fusion via deep graph convolution network for emotion recognition in conversation[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing.PA：ACL，2021：5666-5675.
[125] CERISARA C，JAFARITAZEHJANI S，OLUOKUN A，et al.Multi-task dialog act and sentiment recognition on Mastodon[C]//Proceedings of the 27th International Conference on Computational Linguistics.Santa Fe，PA：ACL，2018：745-754.
[126] LI Y，SU H，SHEN X，et al.DailyDialog：a manually labelled multi-turn dialogue dataset[C]//Proceedings of the Eighth International Joint Conference on Natural Language Processing.Taipei，PA：ACL，2017：986-995.
[127] CHEN S，HSU C，KUO C，et al.EmotionLines：an emotion corpus of multi-party conversations[J].arXiv：1802. 8379，2018.
[128] ZAHIRI S，CHOI J.Emotion detection on TV show transcripts with sequence-based convolutional neural networks[J].arXiv：1708.4299，2017.
[129] CHATTERJEE A，GUPTA U，CHINNAKOTLA M K，et al.Understanding emotions in text using deep learning and big data[J].Computers in Human Behavior，2019，93：309-317.
[130] CHATTERJEE A，NARAHARI K N，JOSHI M，et al.SemEval-2019 task 3：emocontext contextual emotion detection in text[C]//Proceedings of the 13th International Workshop on Semantic Evaluation.Minneapolis，PA：ACL，2019：39-48.
[131] BUSSO C，BULUT M，LEE C，et al.IEMOCAP：interactive emotional dyadic motion capture database[J].Language Resources and Evaluation，2008，42：335-359.
[132] MCKEOWN G，VALSTAR M，COWIE R，et al.The SEMAINE database：annotated multimodal records of emotionally colored conversations between a person and a limited agent[J].IEEE Transactions on Affective Computing，2012，3（1）：5-17.
[133] PORIA S，HAZARIKA D，MAJUMDER N，et al.MELD：a multimodal multi-party dataset for emotion recognition in conversations[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.Florence，PA：ACL，2019：527-536.
[134] SAHA T，PATRA A，SAHA S，et al.Towards emotion-aided multi-modal dialogue act classification[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.PA：ACL，2020：4361-4372.