RoBERTa-Based Sarcasm Detection Model in Conversation Threads from Social Media

doi:10.3778/j.issn.1002-8331.2012-0301

Abstract

Abstract: Sarcasm is a commonly used rhetorical technique in social media, which is wide spread on some social platforms（Twitter, Reddit）, using metaphors and exaggerations to negate, criticize or laugh at people or things. The task of sarcasm detection is essential for understanding people’s actual emotions and beliefs. For sarcasm detection of target text with conversational context, this paper proposes a RoBERTa-based sarcasm detection model in conversation threads from social media. This model is mainly composed of two parts：the former is a feature extraction layer, which uses a more robust transfer learning model RoBERTa to learn features of the conversational context and target text respectively; the latter is a feature fusion layer, since the target text is a response to the context of the conversation, considering that the simple concatenation cannot learn the dialogue relationship between the two very well, this model introduces an improved version of attention-over-attention（AOA） attention module, so that the target text can pay attention to the important information in the conversation context. Finally, experiments are conducted to verify the effectiveness of the model on the public Twitter and Reddit datasets. For the sarcasm detection of the target text, it also analyzes the influence of whether there is a conversation context or not and the number of conversation context utterances on the performance of the model.

Key words: natural language understanding, deep learning, sarcasm detection, transfer learning, attention mechanism, one-dimensional convolution, binary classification

摘要： 讽刺是社会媒体中常用的修辞手法，在一些社交平台上（Twitter、Reddit）普遍存在，用比喻、夸张等方式对人或事进行否定、批评或嘲笑。讽刺检测任务对于理解人们实际的情感和信念至关重要。针对拥有会话上下文的目标文本进行讽刺检测，提出了一种基于RoBERTa的社交媒体会话中的讽刺检测模型。该模型主要由两个部分构成：前者是特征抽取层，采用更加鲁棒的迁移学习模型RoBERTa，对会话上下文文本和目标文本分别进行特征的学习；后者是特征融合层，由于目标文本是对会话上下文的回复，考虑到单纯的级联并不能很好地学习两者之间的对话关系，采用了改进版的attention-over-attention（AOA）注意力模型，使得目标文本可以关注到会话上下文中重要的信息。在公开的Twitter和Reddit两个数据集上进行了实验，验证了模型的有效性，还分析了对于目标文本的讽刺检测，有无会话上下文以及会话上下文数量的多少对模型性能的影响。

关键词: 自然语言处理, 深度学习, 讽刺检测, 迁移学习, 注意力机制, 一维卷积, 二分类

WEI Pengfei, ZENG Bi, LIAO Wenxiong. RoBERTa-Based Sarcasm Detection Model in Conversation Threads from Social Media[J]. Computer Engineering and Applications, 2022, 58(13): 164-170.

魏鹏飞, 曾碧, 廖文雄. 基于RoBERTa的社交媒体会话中的讽刺检测模型[J]. 计算机工程与应用, 2022, 58(13): 164-170.

References

[1] 张林，钱冠群，樊卫国，等.轻型评论的情感分析研究[J].软件学报，2014，25（12）：2790-2807.
ZHANG L，QIAN G Q，FAN W G，et al.Sentiment analysis based on light reviews[J].Journal of Software，2014，25（12）：2790-2807.
[2] MURESAN S，GONZALEZ-IBANEZ R，GHOSH D，et al.Identification of nonliteral language in social media：A case study on sarcasm[J].Journal of the Association for Information Science and Technology，2016，67（11）：2725-2737.
[3] GHOSH D，FABBRI A R，MURESAN S.The role of conversation context for sarcasm detection in online interactions[C]//Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue，2017：186-196.
[4] HAZARIKA D，PORIA S，GORANTLA S，et al.CASCADE：Contextual sarcasm detection in online discussion forums[C]//Proceedings of the 27th International Conference on Computational Linguistics，2018：1837-1848.
[5] OPREA S，MAGDY W.Exploring author context for detecting intended vs perceived sarcasm[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics，2019：2854-2859.
[6] 韩虎，赵启涛，孙天岳，等.面向社交媒体评论的上下文语境讽刺检测模型[J].计算机工程，2021，47（1）：66-71.
HAN H，ZHAO Q T，SUN T Y，et al.Contextual sarcasm detection model for social media comments[J].Computer Engineering，2021，47（1）：66-71.
[7] DAVIDOV D，TSUR O，RAPPOPORT A.Semi-supervised recognition of sarcasm in Twitter and Amazon[C]//Proceedings of the 14th Conference on Computational Natural Language Learning，2010：107-116.
[8] JOSHI A，SHARMA V，BHATTACHARYYA P.Harnessing context incongruity for sarcasm detection[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing，2015：757-762.
[9] PORIA S，CAMBRIA E，HAZARIKA D，et al.A deeper look into sarcastic tweets using deep convolutional neural networks[C]//Proceedings of COLING 2016，the 26th International Conference on Computational Linguistics：Technical Papers，2016：1601-1612.
[10] GHOSH A，VEALE T.Fracking sarcasm using neural network[C]//Proceedings of the 7th Workshop on Computational Approaches to Subjectivity，Sentiment and Social Media Analysis，2016：161-169.
[11] TAY Y，TUAN L A，HUI S C，et al.Reasoning with sarcasm by reading in-between[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics，2018：1010-1020.
[12] ZHANG M，ZHANG Y，FU G.Tweet sarcasm detection using deep neural network[C]//Proceedings of COLING 2016，The 26th International Conference on Computational Linguistics：Technical Papers，2016：2449-2460.
[13] BARUAH A，DAS K，BARBHUIYA F，et al.Context-aware sarcasm detection using BERT[C]//Proceedings of the Second Workshop on Figurative Language Processing，2020：83-87.
[14] DEVLIN J，CHANG M W，LEE K，et al.BERT：Pretraining of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2019：4171-4186.
[15] LIU Y，OTT M，GOYAL N，et al.Roberta：A robustly optimized bert pretraining approach[J].arXiv：1907.11692，2019.
[16] WALLACE B C.Computational irony：A survey and new perspectives[J].Artificial Intelligence Review，2015，43（4）：467-483.
[17] JOSHI A，BHATTACHARYYA P，CARMAN M J.Auto-matic sarcasm detection：A survey[J].ACM Computing Surveys（CSUR），2017，50（5）：1-22.
[18] KHODAK M，SAUNSHI N，VODRAHALLI K.A large self-annotated corpus for sarcasm[C]//Proceedings of the Eleventh International Conference on Language Resources and Evaluation，2018.
[19] MIKOLOV T，CHEN K，CORRADO G，et al.Efficient Estimation of Word Representations in Vector Space[J].arXiv：1301.3781，2013.
[20] PENNINGTON J，SOCHER R，MANNING C D.Glove：Global vectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing，2014：1532-1543.
[21] KIM Y.Convolutional neural networks for sentence classification[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing，2014：1746-1751.
[22] HUANG B，OU Y，CARLEY K M.Aspect level sentiment classification with attention-over-attention neural networks[C]//Proceedings of International Conference on Social Computing，Behavioral?Cultural Modeling and Prediction and Behavior Representation in Modeling and Simulation，2018：197-206.
[23] GHOSH D，FABBRI A R，MURESAN S.Sarcasm analysis using conversation context[J].Computational Linguistics，2018，44（4）：755-792.
[24] JAVDAN S，MINAEI-BIDGOLI B.Applying transformers and aspect-based sentiment analysis approaches on sarcasm detection[C]//Proceedings of the Second Workshop on Figurative Language Processing，2020：67-71.
[25] SRIVASTAVA H，VARSHNEY V，KUMARI S，et al.A novel hierarchical BERT architecture for sarcasm detec-tion[C]//Proceedings of the Second Workshop on Figurative Language Processing，2020：93-97.
[26] DADU T，PANT K.Sarcasm detection using context separators in online discourse[C]//Proceedings of the Second Workshop on Figurative Language Processing，2020：51-55.
[27] KUMAR A，ANAND V.Transformers on sarcasm detection with context[C]//Proceedings of the Second Workshop on Figurative Language Processing，2020：88-92.
[28] KRIZHEVSKY A，SUTSKEVER I，HINTON G E.Imagenet classification with deep convolutional neural networks[J].Communications of the ACM，2017，60（6）：84-90.
[29] SCHMIDHUBER J，HOCHREITER S.Long short-term memory[J].Neural Computer，1997，9（8）：1735-1780.
[30] GHOSH D，VAJPAYEE A，MURESAN S.A report on the 2020 sarcasm detection shared task[J].arXiv：2005. 05814，2020.