Multimodal Fusion Rumor Detection Method Based on Attention Mechanism

doi:10.3778/j.issn.1002-8331.2102-0229

Abstract

Abstract: Rumors have an adverse impact on social life, and online rumors with multiple modalities are more likely to mislead users and spread better than pure text rumors, which makes the detection of multimodal rumors not negligible. At present, the multimodal rumor detection method does not pay attention to the feature fusion between the word and the image area object. Therefore, this paper proposes an attention-based multimodal fusion neural network（AMFNN） for rumor detection. The method carries out high-level information interaction between word and visual object, using the attention mechanism to capture visual features related to the key feature of words. An adaptive attention mechanism（Adaptive-SA） based on the self-attention mechanism is proposed, which restricts the internal information flow by adding auxiliary conditions to make the relationship modeling within the modality is more targeted and diverse. The paper conducts comparative experiments on two datasets about multimodal rumor detection. Experimental results show that, compared with the current related multimodal rumor detection methods, AMFNN can reasonably process multimodal information, thereby improving the accuracy of rumor detection.

Key words: deep learning, attention mechanism, multimodal fusion, rumor detection

摘要： 谣言会对社会生活造成不利影响，同时具有多种模态的网络谣言比纯文字谣言更容易误导用户和传播，这使得对多模态的谣言检测不可忽视。目前关于多模态谣言检测方法没有关注词与图片区域对象之间的特征融合，因此提出了一种基于注意力机制的多模态融合网络AMFNN应用于谣言检测，该方法在词-视觉对象层面进行高级信息交互，利用注意力机制捕捉与关键词语相关的视觉特征；提出了基于自注意力机制的自适应注意力机制Adapive-SA，通过增加辅助条件来约束内部的信息流动，使得模态内的关系建模更有目标性和多样性。在两个多模态谣言检测数据集上进行了对比实验，结果表明，与目前相关的多模态谣言检测方法相比，AMFNN能够合理地处理多模态信息，从而提高了谣言检测的准确性。

关键词: 深度学习, 注意力机制, 多模态融合, 谣言检测

QI Lixin, WAN Shuzhen, TANG Bin, XU Yichun. Multimodal Fusion Rumor Detection Method Based on Attention Mechanism[J]. Computer Engineering and Applications, 2022, 58(19): 209-217.

戚力鑫, 万书振, 唐斌, 徐义春. 基于注意力机制的多模态融合谣言检测方法[J]. 计算机工程与应用, 2022, 58(19): 209-217.

References

[1] KEMP S.Digital around the world in April 2020[EB/OL].（2020-03-30）.https：//wearesocial.com/digital-2020.
[2] ALLCOTT H，GENTZKOW M.Social media and fake news in the 2016 election[J].Journal of Economic Perspectives，2017，31（2）：211-236.
[3] WU K，YANG S，ZHU K Q.False rumors detection on sina weibo by propagation structures[C]//Proceedings of 2015 IEEE 31st International Conference on Data Engineering，April 13-17，2015：651-662.
[4] JIN Z，CAO J，GUO H，et al.Multimodal fusion with recurrent neural networks for rumor detection on microblogs[C]//Proceedings of the 25th ACM International Conference on Multimedia，October 23-27，2017：795-816.
[5] ZAREMBA W，SUTSKEVER I，VINVALS O.Recurrent neural network regularization[J].arXiv：1409.2329，2014.
[6] WANG Y，MA F，JIN Z，et al.EANN：Event adversarial neural networks for multi-modal fake news detection[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining，August 19-23，2018：849-857.
[7] KHATTAR D，GOUD J S，GUPTA M，et al.MVAE：Multimodal variational autoencoder for fake news detection[C]//Proceedings of World Wide Web Conference，May 13-17，2019：2915-2921.
[8] CHUNG J，GULCEHRE C，CHO K，et al.Empirical evaluation of gated recurrent neural networks on sequence modeling[J].arXiv：1412.3555，2014.
[9] DEVLIN J，CHANG M，LEE K，et al.BERT：Pre-training of deep bidirectional transformers for language understanding[J].arXiv：1810.04805，2018.
[10] REN S，HE K，GIRSHICK R，et al.Faster R-CNN：Towards real-time object detection with region proposal networks[J].arXiv：1506.01497，2015.
[11] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[J].arXiv：1706.03762，2017.
[12] ZHU Y，WANG X，ZHONG E，et al.Discovering spammers in social networks[C]//Proceedings of the AAAI Conference on Artificial Intelligence，July 22，2012：171-177.
[13] RUBIN V L，CONROY N，CHEN Y，et al.Fake news or truth? using satirical cues to detect potentially misleading news[C]//Proceedings of the 2nd Workshop on Computational Approaches to Deception Detection，June 17，2016：7-17.
[14] CASTILLO C，MENDOZA M，POBLETE B.Information credibility on twitter[C]//Proceedings of the 20th International Conference on World Wide Web，March 28-April 1，2011：675-684.
[15] QAZVINIAN V，ROSENGREN E，RADEV D，et al.Rumor has it：Identifying misinformation in microblogs[C]//Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing，July 27-31，2011：1589-1599.
[16] KWON S，CHA M，JUNG K，et al.Prominent features of rumor propagation in online social media[C]//Proceedings of the 2013 IEEE 13th International Conference on Data Mining，December 7-13，2013：1103-1108.
[17] WU K，YANG S，ZHU K Q.False rumors detection on sina Weibo by propagation structures[C]//Proceedings of the 2015 IEEE 31st International Conference on Data Engineering，April 13-17，2015：651-662.
[18] MA J，GAO W，MITRA P，et al.Detecting rumors from microblogs with recurrent neural networks[C]//Proceedings of the 25th International Joint Conference on Artificial Intelligence（IJCAI-16），July 9-15，2016：3818-3824.
[19] CHEN T，LI X，YIN H，et al.Call attention to rumors：Deep attention based recurrent neural networks for early rumor detection[C]//Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining，June 3-6，2018：40-52.
[20] YU F，LIU Q，WU S，et al.A convolutional approach for misinformation identification[C]//Proceedings of the 26th International Joint Conference on Artificial Intelligence，August 19-25，2017：3901-3907.
[21] MA J，GAO W，WONG K.Rumor detection on twitter with tree-structured recursive neural networks[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics，July 15-20，2018：1980-1989.
[22] ZHANG J，DONG B，PHILIP S Y.Deep diffusive neural network based fake news detection from heterogeneous social networks[C]//Proceedings of the 2019 IEEE International Conference on Big Data（Big Data），December 9-12，2019：1259-1266.
[23] LIU Y，WU Y F.Early detection of fake news on social media through propagation path classification with recurrent and convolutional networks[C]//Proceedings of the AAAI Conference on Artificial Intelligence，February 2-7，2018：354-361.
[24] JIN Z，CAO J，ZHANG Y，et al.Novel visual and statistical image features for microblogs news verification[J].IEEE Transactions on Multimedia，2016，19（3）：598-608.
[25] ZHANG T，WANG D，CHEN H，et al.BDANN：BERT-based domain adaptation neural network for multi-modal fake news detection[C]//Proceedings of the 2020 International Joint Conference on Neural Networks（IJCNN），July 19-24，2020：1-8.
[26] ZADEH A，CHEN M，PORIA S，et al.Tensor fusion network for multimodal sentiment analysis[J].arXiv：1707. 07250，2017.
[27] HOU M，TANG J，ZHANG J，et al.Deep multimodal multilinear fusion with high-order polynomial pooling[C]//Advances in Neural Information Processing Systems，2019：12136-12145.
[28] XU N，MAO W，CHEN G.Multi-interactive memory network for aspect based multimodal sentiment analysis[C]//Proceedings of the AAAI Conference on Artificial Intelligence，Jan 27-Feb 1，2019：371-378.
[29] PENNINGTON J，SOCHER R，MANNING C D.Glove：Global vectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing（EMNLP），October 25-29，2014：1532-1543.
[30] GAO P，JIANG Z，YOU H，et al.Dynamic fusion with intra-and inter-modality attention flow for visual question answering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，June 15-20，2019：6639-6648.
[31] BOIDIDOU C，ANDREADOU K，PAPADOPOULOS S，et al.Verifying multimedia use at media Eval 2015[C]//Proceedings of the MediaEval 2015 Multimedia Benchmark Workshop，September 14-15，2015.
[32] ANTOL S，AGRAWAL A，LU J，et al.VQA：Visual question answering[C]//Proceedings of the IEEE International Conference on Computer Vision，December 7-13，2015：2425-2433.
[33] VINYALS O，TOSHEY A，BENGIO S，et al.Show and tell：A neural image caption generator[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，June 7-12，2015：3156-3164.
[34] LIN T Y，GOYAL P，GIRSHICK R，et al.Focal loss for dense object detection[C]//Proceedings of the IEEE International Conference on Computer Vision，October 22-29，2017：2980-2988.
[35] GODBOLE S，SARAWAGI S.Discriminative methods for multi-labeled classification[C]//Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining，May 26-28，2004：22-30.