Survey of Deep Learning-Based on Emotion Generation in Conversation

doi:10.3778/j.issn.1002-8331.2304-0131

Abstract

Abstract: Emotion generation is a subtask in the study of artificial affective computing, where the emotion generation task in a dialog system aims to generate emotion categories in the discourse to be replied to. Conversational emotion generation aims to generate emotion categories in the discourse to be replied, which can promote the research of conversational emotion understanding and conversational expression, and also has important theoretical significance and practical application value in many intelligent fields such as intelligent gossip bots, emotional comfort, recommendation systems and human-computer emotional interaction. Thanks to the excellent performance of deep neural networks in the field of natural language processing, deep learning-based emotion generation for conversational systems has received more and more attention from researchers. To summarize the current work related to deep learning-based conversational emotion generation, the current research on emotion generation for conversational systems using deep learning mainly contains three aspects：emotion perception, emotion prediction and emotion decision making. Some commonly used emotion dialogue datasets are briefly introduced, and finally, an overview of current issues in this task is summarized and future trends are prospected.

Key words: emotion generation in conversation, artificial emotions, deep learning

摘要： 情绪生成是人工情感计算研究中的子任务，在对话系统中情绪生成任务旨在生成待回复话语中的情绪类别。对话情绪生成可以推动对话情绪理解和对话表达研究，同时在智能闲聊机器人、情绪安慰、推荐系统和人机情感交互等诸多智能化领域具有重要的理论意义和实际应用价值。得益于深度神经网络在自然语言处理领域的优异表现，基于深度学习的对话系统情绪生成受到越来越多研究人员的关注。总结目前基于深度学习的对话情绪生成相关工作，现阶段利用深度学习的对话系统情绪生成相关研究主要包含三方面内容：情绪感知、情绪预测和情绪决策。简要介绍了一些常用的情绪对话数据集，最后对该任务当前问题进行了归纳概况并展望未来发展趋势。

关键词: 对话情绪生成, 人工情感, 深度学习

ZHOU Yutong, MA Zhiqiang, XU Biqi, JIA Wenchao, LYU Kai, LIU Jia. Survey of Deep Learning-Based on Emotion Generation in Conversation[J]. Computer Engineering and Applications, 2024, 60(7): 13-25.

周钰童, 马志强, 许璧麒, 贾文超, 吕凯, 刘佳. 基于深度学习的对话情绪生成研究综述[J]. 计算机工程与应用, 2024, 60(7): 13-25.

References

[1] MARIETTO M, AGUIAR R, BARBOSA G, et al. Artificial intelligence markup language: a brief tutorial[J]. arXiv:1307.
3091,2013.
[2] PRENDINGER H, MORI J, ISHIZUKA M. Using human physiology to evaluate subtle expressivity of a virtual quizmaster in a mathematical game[J]. International Journal of Human-Computer Studies, 2005, 62(2): 231-245.
[3] 陈晓婷, 李实. 对话情绪识别综述[J]. 计算机工程与应用, 2023, 59(3): 33-48.
CHEN X T, LI S. Survey on emotion recognition in conversation[J]. Computer Engineering and Applications, 2023, 59(3): 33-48.
[4] ZHOU H, HUANG M, ZHANG T, et al. Emotional chatting machine: emotional conversation generation with internal and external memory[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
[5] WANG K, WAN X, SENTIGA N, et al. Generating sentimental texts via mixture adversarial networks[C]//Proceedings of IJCAI, 2018: 4446-4452.
[6] PICARD R W. Affective computing[M]. London, England: MIT Press, 1997.
[7] 王志良. 人工心理与人工情感[J]. 智能系统学报, 2006, 1(1): 38-43.
WANG Z L. Artificial psychology and artificial emotion[J]. CAAI Transactions on Intelligent Systems, 2006, 1(1): 38-43.
[8] MUNEZERO M D, MONTERO C S, SUTINEN E, et al. Are they different? Affect, feeling, emotion, sentiment, and opinion detection in text[J]. IEEE Transactions on Affective Computing, 2014, 5(2): 101-111.
[9] CHAO C H, LUN W K. SocialNLP 2018 EmotionX challenge overview: recognizing emotions in dialogues[C]//Proceedings of the Sixth International Workshop on Natural Language Processing for Social Media, Melbourne, Australia, 2018: 27-31.
[10] JIAO W X, YANG H Q, KING I, et al. HiGRU: hierarchical gated recurrent units for utterance-level emotion recognition[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, 2019: 397-406.
[11] JIE C, MICHAEL T, ZAC I, et al. ?Observing dialogue in therapy: categorizing and forecasting behavioral codes[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 2019: 5599-5611.
[12] ZHANG D, WU L, SUN C L, et al. Modeling both context- and speaker-sensitive dependence for emotion detection in multi-speaker conversations[C]//Proceedings of the International Joint Conference on Artificial Intelligence, 2019.
[13] BU J H, REN L, ZHENG S, et al. ASAP: a Chinese review dataset towards aspect category sentiment analysis and rating prediction[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021: 2069-2079.
[14] LI Y R, LI K, NING H K, et al. Towards an online empathetic chatbot with emotion causes[C]//Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’21), New York, NY, USA, 2021: 2041-2045.
[15] TIAN H, GAO C, XIAO X Y, et al. ?SKEP: sentiment knowledge enhanced pre-training for sentiment analysis[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 4067-4076.
[16] 赵妍妍, 陆鑫, 赵伟翔, 等. 情感对话技术综述[J/OL]. 软件学报: 1-26[2023-06-05]. https://doi.org/10.13328/j.cnki.jos.
006807.
ZHAO Y Y, LU X, ZHAO W X, et al. Survey on emotional dialogue techniques[J/OL]. Journal of Software: 1-26[2023-06-05]. https://doi.org/10.13328/j.cnki.jos.006807.
[17] MAYER J D. What is emotional intelligence?[Z]. 2004.
[18] COLOMBO P, WITON W, MODI A, et al. Affect-driven dialog generation[J]. arXiv:1904.02793, 2019.
[19] MA Y, NGUYEN K L, XING F Z, et al. A survey on empathetic dialogue systems[J]. Information Fusion, 2020, 64: 50-70.
[20] MAO Y, CAI F, GUO Y, et al. Incorporating emotion for response generation in multi-turn dialogues[J]. Applied Intelligence, 2022, 52(7): 7218-7229.
[21] PORIA S, CAMBRIA E, HAZARIKA D, et al. Context-dependent sentiment analysis in user-generated videos[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Jul 31-Aug 4, 2017. Stroudsburg, PA: ACL, 2017: 873-883.
[22] HSU C C, CHEN S Y, KUO C C, et al. EmotionLines: an emotion corpus of multi-party conversations[C]//Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, 2018. ?
[23] MA D, LI S, ZHANG X, et al. Interactive attention networks for aspect-level sentiment classification[J]. arXiv:1709.00893,2017.
[24] LI W, SHAO W, JI S, et al. BiERU: bidirectional emotional recurrent unit for conversational sentiment analysis[J]. Neurocomputing, 2022, 467: 73-82.
[25] BOWMAN S R, VILNIS L, VINYALS O, et al. Generating sentences from a continuous space[J]. arXiv:1511.06349, 2015.
[26] IULIAN S, ALESSANDRO S, RYAN L, et al. A hierarchical latent variable encoder-decoder model for generating dialogues[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2017: 3295-3301.
[27] XIANG B, ZHOU L. Improving twitter sentiment analysis with topic-based mixture modeling and semi-supervised training[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2014: 434-439.
[28] RANA T A, CHEAH Y N, LETCHMUNAN S. Topic modeling in sentiment analysis: a systematic review[J]. Journal of ICT Research & Applications, 2016, 10(1): 76-93.
[29] REN Y, ZHANG Y, ZHANG M, et al. Improving twitter sentiment classification using topic-enriched multi-prototype word embeddings[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2016.
[30] 王建成, 徐扬, 刘启元, 等. 基于神经主题模型的对话情感分析[J]. 中文信息学报, 2020, 34(1): 106-112.
WANG J C, XU Y, LIU Q Y, et al. Dialog sentiment analysis with neural topic model[J]. Journal of Chinese Information Processing, 2020, 34(1): 106-112.
[31] ZHU L, PERGOLA G, GUI L, et al. Topic-driven and knowledge-aware transformer for dialogue emotion detection[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume1: Long Papers), 2021.
[32] HAZARIKA R, ?PORIA S, ?MIHALCEA R, et al. ?ICON: interactive conversational memory network for multimodal emotion detection[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, 2018: 2594-2604.
[33] MAJUMDER N, PORIA S, HAZARIKA D, et al. Dialoguernn: an attentive RNN for emotion detection in conversations[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2019: 6818-6825.
[34] GHOSAL D, ?MAJUMDER N, PORIA S, et al. DialogueGCN: a graph convolutional neural network for emotion recognition in conversation[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 2019: 154-164.
[35] ZHANG D, WU L, SUN C, et al. Modeling both context-and speaker-sensitive dependence for emotion detection in multi-speaker conversations[C]//Proceedings of IJCAI, 2019: 5415-5421.
[36] HU D, WEI L, HUAI X, et al. DialogueCRN: contextual reasoning networks for emotion recognition in conversations[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021.
[37] CERISARA C, JAFARITAZEHJANI S, OLUOKUN A, et al. Multi-task dialog act and sentiment recognition on Mastodon[C]//Proceedings of the 27th International Conference on Computational Linguistics, Santa Fe, New Mexico, Aug 20-26, 2018. Stroudsburg, PA: ACL, 2018: 745-754.
[38] KIM M, KIM H. Integrated neural network model for identifying speech acts, predicators, and sentiments of dialogue utterances[J]. Pattern Recognition Letters, 2018, 101: 1-5.
[39] LI J, FEI H, JI D H. Modeling local contexts for joint dialogue act recognition and sentiment classification with bi-channel dynamic convolutions[C]//Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain, 2020: 616-626.
[40] QIN L, CHE W, LI Y, et al. DCR-Net: a deep co-interactive relation network for joint dialog act recognition and sentiment classification[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2020: 8665-8672.
[41] QIN L, LI Z, CHE W, et al. Co-GAT: a co-interactive graph attention network for joint dialog act recognition and sentiment classification[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2021: 13709-13717.
[42] XING B W, TSANG I. DARER: dual-task temporal relational recurrent reasoning network for joint dialog sentiment classification and act recognition[C]//Findings of the Association for Computational Linguistics (ACL 2022), Dublin, Ireland, 2022: 3611-3621.
[43] CAO J, TANANA M, IMEL Z E, et al. Observing dialogue in therapy: categorizing and forecasting behavioral codes[J]. arXiv:1907.00326, 2019.
[44] BOSSELUT A, RASHKIN H, SAP M, et al. COMET: commonsense transformers for automatic knowledge graph construction[J]. arXiv:1906.05317, 2019.
[45] MATTHEW P, MARK N. Deep contextualized word representations[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018, 2227-2237.
[46] ZHOU H, YOUNG T. Commonsense knowledge aware conversation generation with graph attention[C]//Proceedings of the 27th International Joint Conference on Artificial Intelligence, 2018: 4623-4629.
[47] ROBYN S, JOSHUA C. ConceptNet 5.5: an open multilingual graph of general knowledge[C]//Proceedings of the 31st AAAI Conference on Artificial Intelligence, 2017: 4444-4451.
[48] MAARTEN S, RONAN L. ATOMIC: an atlas of machine commonsense for if-then reasoning[C]//Proceedings of the 33rd AAAI Conference on Artificial Intelligence, 2019: 3027-3035.
[49] TOMAS M I, CHEN K. Efficient estimation of word representations in vector space[C]//Proceedings of the 1st International Conference on Learning Representations, 2013.
[50] HANNAH R, MAARTEN S, EMILY A, et al. event2mind: commonsense inference on events, intents, and reactions[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018: 463-473.
[51] JEFFREY P, RICHARD S, CHRISTOPHER M. GloVe: global vectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, 2014: 1532-1543.
[52] GHOSAL D, MAJUMDER N, GELBUKH A, et al. Cosmic: commonsense knowledge for emotion identification in conversations[J]. arXiv:2010.02795, 2020.
[53] ZHONG P X, WANG D. Knowledge-enriched transformer for emotion detection in textual conversations[C]//Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, 2019: 165-176.
[54] KINGMA D P, WELLING M. Auto-encoding variational bayes[J]. arXiv:1312.6114, 2013.
[55] LI J N, LIN Z, FU P, et al. Past, present, and future: conversational emotion recognition through structural modeling of psychological knowledge[C]//Findings of the Association for Computational Linguistics (EMNLP 2021), Punta Cana, Dominican Republic, 2021: 1204-1214.
[56] LI D, ZHU X, LI Y, et al. Enhancing emotion inference in conversations with commonsense knowledge[J]. Knowledge-Based Systems, 2021, 232: 107449.
[57] LI W, ZHU L, MAO R, et al. SKIER: a symbolic knowledge integrated model for conversational emotion recognition[C]//Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, 2023.
[58] ZHAO W X, ZHOU K, LI J, et al. A survey of large language models[J]. arXiv:2303.18223,2023.
[59] BAO S, HE H, WANG F, et al. PLATO-XL: exploring the large-scale pre-training of dialogue generation[J]. arXiv:2109.
09519, 2021.
[60] HASEGAWA T, ?KAJI N. Predicting and eliciting addressee’s emotion in online dialogue[C]//Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, 2013: 964-972.
[61] BOTHE C, SVEN M. Dialogue based neural learning to estimate the sentiment of a next upcoming utterance[C]//Proceedings of the 26th International Conference on Artificial Neural Networks, 2017: 477-485.
[62] WANG Z Q, ZHU X J, ZHANG Y, et al. Sentiment forecasting in dialog[C]//Proceedings of the 28th International Conference on Computational Linguistics, 2020: 2448-2458.
[63] RASHKIN H, MICHAEL S. Towards empathetic open-domain conversation models: a new benchmark and dataset[C]//Proceedings of the 57th annual meeting of the Association for Computational Linguistics, 2019.
[64] HANNAH R, ANTOINE B, MAARTEN S, et al. Modeling naive psychology of characters in simple commonsense stories[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, 2018: 2289-2299.
[65] GAONKER R, ?KWON H. Modeling label semantics for predicting emotional reactions[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 4687-4692.
[66] KIM J, YOO J, LIM H, et al. Sentiment prediction using collaborative filtering[C]//Proceedings of the International AAAI Conference on Web and Social Media, ?2021: 685-688.
[67] THAVAREESAN S, MAHESAN S. Sentiment lexicon expansion using Word2vec and fastText for sentiment prediction in Tamil texts[C]//2020 Moratuwa Engineering Research Conference (MERCon), Moratuwa, Sri Lanka, 2020: 272-276.
[68] MIKOLOV T, CHEN K, CORRADO G, et al. Efficient estimation of word representations in vector space[J]. arXiv:1301.3781, 2013.
[69] ARMAND J, EDOUARD G, PIOTR B, et al. Bag of tricks for efficient text classification[C]//Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2, Short Papers), Valencia, Spain, 2017: 427-431.
[70] PATRICK H, GIUSEPPE C. ?From sentiment annotations to sentiment prediction through discourse augmentation[C]//Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain, 2020: 185-197.
[71] WANG X, JI H Y, SHI C, et al. Heterogeneous graph attention network[C]//The World Wide Web Conference (WWW’19), New York, NY, 2019.
[72] TAI K S, SOCHER R, MANNING C D. Improved semantic representations from tree-structured long short-term memory networks[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Beijing, China, 2015: 1556-1566.
[73] LI D Y, LI Y, WANG S G. Interactive double states emotion cell model for textual dialogue emotion prediction[J]. Knowledge-Based Systems, 2020, 189: 105084.
[74] CHANG Y C, HSING Y C. Emotion-infused deep neural network for emotionally resonant conversation[J]. Applied Soft Computing, 2021, 113: 107861.
[75] YANG Z, ASYROFI M H, LO D. BiasRV: uncovering biased sentiment predictions at runtime[C]//Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2021: 1540-1544.
[76] YANG H , SHEN J. Emotion dynamics modeling via BERT[C]//Proceedings of the International Conference on Neural Networks (IJCNN 2021), 2021.
[77] CHEN Z, SONG R , XIE X , et al. Neural response generation with relevant emotions for short text conversation[M]. Cham: Springer, 2019.
[78] WEI W, LIU J, MAO X, et al. Emotion-aware chat machine: for human-like emotional interaction[C]//Proceedings of the 28th ACM International Conference, 2019.
[79] ZHOU L, GAO J, D LI, et al. The design and implementation of Xiaoice, an empathetic social chatbot[J]. Computational Linguistics, 2020, 46(1): 1-62.
[80] GHOSH S, CHOLLET M, LAKSANA E, et al. Affect-LM: a neural language model for customizable affective text generation[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Jul 30- Aug 4, 2017. Stroudsburg, PA: ACL, 2017: 634-642.
[81] ASGHAR N, POUPART P, HOEY J, et al. Affective neural response generation[C]//Proceedings of the European Conference on Information Retrieval. Cham: Springer, 2018: 154-166.
[82] 裴正蒙. 情绪引导式情感对话系统研究[D]. 合肥: 合肥工业大学, 2020.
PEI Z M. Research on emotion-guided emotion dialogue system[D]. Hefei: Hefei University of Technology, 2020.
[83] WEN Z, CAO J, YANG R, et al. Automatically select emotion for response via personality affected emotion transition[C]//The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021), 2021.
[84] YANG L, SHEN Y, MAO Y, et al. Hybrid curriculum learning for emotion recognition in conversation[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2022: 11595-11603.
[85] BUSSO C, BULUT M, LEE C, ?et al. ?IEMOCAP: interactive emotional dyadic motion capture database[J]. Language Resources & Evaluation, ?2008, 42: 335-359.
[86] LI Y R, SU H, SHEN X Y, et al. DailyDialog: a manually labelled multi-turn dialogue dataset[C]//Proceedings of the 8th International Joint Conference on Natural Language Processing, 2017: 986-995.
[87] ZAHIRI S M, CHOI J D. Emotion detection on TV show transcripts with sequence-based convolutional neural networks[J]. arXiv:1708.04299, 2017.
[88] SOUJANYA P, DEVAMANYU H, NAVONIL M, et al. MELD: a multimodal multi-party dataset for emotion recognition in conversations[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, 2019: 527-536.
[89] LIU S Y, ZHENG C J. Towards emotional support dialog systems[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, 2021.
[90] WANG H, LU Z D, LI H, et al. A dataset for research on short-text conversations[C]//Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Washington, USA, 2013: 935-945.
[91] WEN Z Y, CAO J N, YANG R S, et al. ?Automatically select emotion for response via personality-affected emotion transition[C]//Findings of the Association for Computational Linguistics (ACL-IJCNLP 2021), 2021: 5010-5020.
[92] CHEN Y, FAN W, XING X, et al. CPED: a large-scale Chinese personalized and emotional dialogue dataset for conversational AI[J]. arXiv:2205.14727, 2022.
[93] COSTA P T, MCCRAE R R. Normal personality assessment in clinical practice: the NEO personality inventory[J]. Psychological Assessment, 1992, 4(1): 5.
[94] DALE R. GPT-3: what’s it good for?[J]. Natural Language Engineering, 2021, 27(1): 113-118.