Multimodal Feature Adaptive Fusion for Fake News Detection

doi:10.3778/j.issn.1002-8331.2303-0316

Abstract

Abstract: In order to solve the problem that it is difficult to make full use of graphic and text information in multimodal news detection in social media news and to explore efficient multimodal information interaction methods, an adaptive multimodal feature fusion model for fake news detection is proposed. First, the model extracts and represents news text semantic features, text emotional features, and image-text semantic difference features; then, weighted splicing and fusion of various features are performed by adding adaptive weight parameters to reduce the redundancy introduced by model splicing; finally, the fusion feature is sent to the classifier. Experimental results show that the proposed model outperforms the current state-of-the-art models in evaluation indicators such as F1 score. It effectively improves the performance of fake news detection and provides strong support for the detection of fake news in social media.

Key words: fake news detection, emotional feature, image caption, adaptive fusion

摘要： 为解决社交媒体新闻中多模态新闻检测难以充分利用图文信息问题以及探索高效的多模态信息交互方法，提出了一种多模态特征自适应融合的虚假新闻检测模型。分别对新闻文本语义特征、文本情感特征和图文语义差异特征进行提取和表示；通过添加自适应权重参数的方式对多种特征进行加权拼接融合，以减少模型拼接时引入的冗余信息；将融合特征送入分类器中进行新闻的真假分类。实验结果表明，所提出的模型在F1值等评价指标上都优于当前先进的模型。有效提升了虚假新闻检测性能，为社交媒体中虚假新闻的检测提供了有力支持。

关键词: 虚假新闻检测, 情感特征, 图像描述, 自适应融合

WANG Teng, ZHANG Dawei, WANG Liqin, DONG Yongfeng. Multimodal Feature Adaptive Fusion for Fake News Detection[J]. Computer Engineering and Applications, 2024, 60(13): 102-112.

王腾, 张大伟, 王利琴, 董永峰. 多模态特征自适应融合的虚假新闻检测[J]. 计算机工程与应用, 2024, 60(13): 102-112.

References

[1] 中国互联网络信息中心. 第50次中国互联网络发展状况统计报告[R]. 北京: 中国互联网络信息中心, 2022.
China Internet Network Information Center. The 50th statistical report on China’s Internet development[R]. Beijing: China Internet Network Information Center, 2022.
[2] MEEL P, VISHWAKARMA D K. Fake news, rumor, information pollution in social media and web: a contemporary survey of state-of-the-arts, challenges and opportunities[J]. Expert Systems with Applications, 2019, 153(1): 112986.
[3] SATU M S, KHAN M I, MAHMUD M, et al. TClustVID: a novel machine learning classification model to investigate topics and sentiment in COVID-19 tweets[J]. Knowledge-Based Systems, 2021, 226: 107126.
[4] RISH I. An empirical study of the Naive Bayes classifier[J]. Journal of Universal Computer Science, 2001, 1(2): 127.
[5] SAFAVIAN S R, LANDGREBE D. A survey of decision tree classifier methodology[J]. IEEE Transactions on Systems, Man, and Cybernetics, 1991, 21(3): 660-674.
[6] NOBLE W S. What is a support vector machine?[J]. Nature Biotechnology, 2006, 24(12): 1565-1567.
[7] VOLKOVA S, SHAFFER K, JIN Y J, et al. Separating facts from fiction: linguistic models to classify suspicious and trusted news posts on twitter[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2017: 647-653.
[8] POTTHAST M, KIESEL J, REINARTZ K, et al. A stylometric inquiry into hyperpartisan and fake news[J]. arXiv:1702.
05638, 2017.
[9] CASTILLO C, MENDOZA M, POBLETE B. Information credibility on twitter[C]//Proceedings of the 20th International Conference on World Wide Web, 2011: 675-684.
[10] CHEN Y, CONROY N J, RUBIN V L. Misleading online content: recognizing clickbait as false news[C]//Proceedings of the 2015 ACM on Workshop on Multimodal Deception Detection, 2015: 15-19.
[11] FENG S, BANERJEE R, CHOI Y. Syntactic stylometry for deception detection[C]//Proceedings of the Meeting of the Association for Computational Linguistics: Short Papers, 2012: 171-175.
[12] MA J, GAO W, MITRA P, et al. Detecting rumors from microblogs with recurrent neural networks[C]//Proceedings of the International Joint Conference on Artificial Intelligence, 2016.
[13] YU F, LIU Q, WU S, et al. A convolutional approach for misinformation identification[C]//Twenty-Sixth International Joint Conference on Artificial Intelligence, 2017: 901-907.
[14] CHEN T, LI X, YIN H, et al. Call attention to rumors: deep attention based recurrent neural networks for early rumor detection[C]//Lecture Notes in Computer Science: Trends and Applications in Knowledge Discovery and Data Mining PAKDD 2018 Workshops, Melbourne, VIC, 2018: 40-52
[15] 沈瑞琳, 潘伟民, 彭成, 等. 基于多任务学习的微博谣言检测方法[J]. 计算机工程与应用, 2021, 57(24): 192-197.
SHEN R L, PAN W M, PENG C, et al. Microblog rumor detection method based on multi-task learning[J]. Computer Engineering and Applications, 2021, 57(24): 192-197.
[16] JIN Z, CAO J , ZHANG Y, et al. Novel visual and statistical image features for microblogs news verification[J]. IEEE Transactions on Multimedia, 2017, 19(3): 598-608.
[17] KE W, SONG Y, ZHU K Q. False rumors detection on Sina Weibo by propagation structures[C]//Proceedings of the IEEE International Conference on Data Engineering, 2015: 651-662.
[18] QI P, CAO J, YANG T, et al. Exploiting multi-domain visual information for fake news detection[C]//Proceedings of the IEEE International Conference on Data Mining, 2019: 518-527.
[19] CAO J, QI P, SHENG Q, et al. Exploring the role of visual content in fake news detection[M]//Disinformation, misinformation, and fake news in social media: emerging research challenges and opportunities. Cham: Springer, 2020: 141-161.
[20] JIN Z, CAO J, GUO H, et al. Multimodal fusion with recurrent neural networks for rumor detection on microblogs[C]//Proceedings of the 25th ACM International Conference on Multimedia, 2017: 795-816.
[21] KUMARI R, EKBAL A. AMFB: attention based multimodal factorized bilinear pooling for multimodal fake news detection[J]. Expert Systems with Application, 2021, 184: 115412.
[22] WANG Y, MA F, WANG H, et al. Multimodal emergent fake news detection via meta neural process networks[C]//Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021: 3708-3716.
[23] TUAN N M D, MINH P Q N. Multimodal fusion with BERT and attention mechanism for fake news detection[C]//Proceedings of the 2021 RIVF International Conference on Computing and Communication Technologies, 2021: 1-6.
[24] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, 2019: 4171-4186.
[25] WANG Y, MA F, JIN Z, et al. EANN: event adversarial neural networks for multi-modal fake news detection[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018: 849-857.
[26] ZHUANG Y, ZHANG Y. Yet at memotion 2.0 2022: hate speech detection combining bilstm and fully connected layers[C]//Proceedings of De-Factify: Workshop on Multimodal Fact Checking and Hate Speech Detection, 2022.
[27] PENNINGTON J, SOCHER R, MANNING C D. Glove: global vectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014: 1532-1543.
[28] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv:1409.
1556, 2014.
[29] KHATTAR D, GOUD J S, GUPTA M, et al. MVAE: multimodal variational autoencoder for fake news detection[C]//Proceedings of the The World Wide Web Conference, 2019: 2915-2921.
[30] QIAN S, WANG J, HU J, et al. Hierarchical multi-modal contextual attention network for fake news detection[C]// Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021: 153-162.
[31] LI P, SUN X, YU H, et al. Entity-oriented multi-modal alignment and fusion network for fake news detection[J]. IEEE Transactions on Multimedia, 2021, 24: 3455-3468.
[32] ZHOU X, WU J, ZAFARANI R. SAFE: similarity-aware multi-modal fake news detection[C]//Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, 2020: 354-367.
[33] TIAN H, GAO C, XIAO X, et al. SKEP: sentiment knowledge enhanced pre-training for sentiment analysis[J]. arXiv:2005.05635, 2020.
[34] SHEN D, ZHANG X, HENAO R, et al. Improved semantic-aware network embedding with fine-grained word alignment[J]. arXiv:1808.09633, 2018.
[35] NAN Q, CAO J, ZHU Y, et al. MDFEND: multi-domain fake news detection[C]//Proceedings of the 30th ACM International Conference on Information & Knowledge Management, 2021: 3343-3347.