Fake News Detection in Multi-Domain and Multi-Modal Fusion Networks

doi:10.3778/j.issn.1002-8331.2403-0252

Abstract

Abstract: The public is able to quickly obtain massive amounts of valuable information from the Internet, but it also facilitates the widespread dissemination of fake news. Therefore, it becomes very important to find and mark out fake news on social media, and the fast and accurate identification of fake news can effectively prevent the formation of negative online public opinion and reduce the adverse social impact. On the basis of the existing fake news recognition research, a multi-domain and multi-modal fusion network (DMMFN) for fake news detection is constructed. In the DMMFN model, the BERT model is used to convert the text content of the fake news into text vectors, and the CLIP is used to extract the feature information of the images. By considering the correlation and interaction between text and images, a multimodal fusion network is established. Two combined matrices are formed to promote information interaction and fusion between different modalities. A multi-domain classification is introduced so that multi-modal features of different events can be mapped to the same feature space. The performance of this model is tested on Twitter and Weibo datasets, and the experimental results demonstrate that the DMMFN model outperforms baseline models such as SIMPLE and CCD in terms of accuracy, precision and F1 scores.

Key words: fake news, BERT, CLIP, multimodal fusion, multi-domain classification

摘要： 公众能够从互联网快速获取海量有价值的信息，但也为虚假新闻的广泛和迅速传播提供了便利。因此，在社交媒体上发现并标记出虚假新闻变得非常重要，快速准确地识别出虚假新闻能够有效防止负面网络舆情的形成，减少不良社会影响。在现有虚假新闻识别研究基础上，构建了多领域多模态融合网络虚假新闻检测模型（DMMFN）。DMMFN模型中使用了BERT模型将虚假新闻的文本内容转换为文本向量，使用CLIP提取图片特征信息，考虑文本与图像相关性与交互性，建立一个多模态融合网络，组成的两个联合矩阵促进不同模态之间的信息交互和融合。引入一个多领域分类器，让不同事件的多模态特征可以映射到同一个特征空间中。在Twitter和Weibo数据集中测试了模型的性能，实验结果证明，DMMFN模型在accuracy、precision和F1分数上均优于SIMPLE、CCD等基线模型。

关键词: 虚假新闻, BERT, CLIP, 多模态融合, 多领域分类

JIAO Shiming, YU Kai. Fake News Detection in Multi-Domain and Multi-Modal Fusion Networks[J]. Computer Engineering and Applications, 2025, 61(11): 238-248.

焦世明, 于凯. 多领域多模态融合网络的虚假新闻检测[J]. 计算机工程与应用, 2025, 61(11): 238-248.

References

[1] BONDIELLI A, MARCELLONI F. A survey on fake news and rumour detection techniques[J]. Information Sciences, 2019, 497: 38-55.
[2] SINGH V K, GHOSH I, SONAGARA D. Detecting fake news stories via multimodal analysis[J]. Journal of the Association for Information Science and Technology, 2021, 72(1): 3-17.
[3] IBRAHIM E, FATTOH D, ALI F, et al. Fake news detection based on word and document embedding using machine learning classifiers[C]//Proceedings of the 2nd International Workshop on Human-Centric Smart Environments for Health and Well-Being, 2021: 125-130.
[4] AHMAD I, YOUSAF M, YOUSAF S, et al. Fake news detection using machine learning ensemble methods[J]. Complexity, 2020, 2020: 8885861.
[5] PIERRI F, CERI S. False news on social media[J]. ACM SIGMOD Record, 2019, 48(2): 18-27.
[6] NGADIRON S, AZIZ A A, MOHAMED S S. The spread of covid-19 fake news on social media and its impact among malaysians[J]. International Journal of Law, Government and Communication, 2021, 6(22): 253-260.
[7] SEGURA-BEDMAR I, ALONSO-BARTOLOME S. Multimodal fake news detection[J]. Information, 2022, 13(6): 284.
[8] ZHANG T, WANG D, CHEN H H, et al. BDANN: BERT-based domain adaptation neural network for multi-modal fake news detection[C]//Proceedings of the International Joint Conference on Neural Networks. Piscataway: IEEE, 2020: 1-8.
[9] LI S, YAO T, LI S, et al. Semantic‐enhanced multimodal fusion network for fake news detection[J]. International Journal of Intelligent Systems, 2022, 37(12): 12235-12251.
[10] REN Y F, JI D H. Neural networks for deceptive opinion spam detection: an empirical study[J]. Information Sciences, 2017, 385: 213-224.
[11] LIU C, WU X H, YU M, et al. A two-stage model based on BERT for short fake news detection[C]//Proceedings of the 12th International Conference on Knowledge Science, Engineering and Management, 2019: 172-183.
[12] SEPúLVEDA-TORRES R, VICENTE M, SAQUETE E, et al. HeadlineStanceChecker: exploiting summarization to detect headline disinformation[J]. Journal of Web Semantics, 2021, 71: 100660.
[13] AMAAR A, ALJEDAANI W, RUSTAM F, et al. Detection of fake job postings by utilizing machine learning and natural language processing approaches[J]. Neural Processing Letters, 2022, 54(3): 2219-2247.
[14] HUANG Y Q, GAO M, WANG J, et al. Meta-prompt based learning for low-resource false information detection[J]. Information Processing & Management, 2023, 60(3): 103279.
[15] QI P, CAO J, YANG T Y, et al. Exploiting multi-domain visual information for fake news detection[C]//Proceedings of the IEEE International Conference on Data Mining. Piscataway: IEEE, 2019: 518-527.
[16] JIN Z W, CAO J, GUO H, et al. Multimodal fusion with recurrent neural networks for rumor detection on microblogs[C]//Proceedings of the 25th ACM International Conference on Multimedia. New York: ACM, 2017: 795-816.
[17] SINGHAL S, SHAH R R, CHAKRABORTY T, et al. SpotFake: a multi-modal framework for fake news detection[C]//Proceedings of the IEEE 5th International Conference on Multimedia Big Data. Piscataway: IEEE, 2019: 39-47.
[18] WANG Y Q, MA F L, JIN Z W, et al. EANN: event adversarial neural networks for multi-modal fake news detection[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. New York: ACM, 2018: 849-857.
[19] LIU H, WANG W, LI H. Towards multi-modal sarcasm detection via hierarchical congruity modeling with knowledge enhancement[J]. arXiv:2210.03501, 2022.
[20] SONG C G, NING N W, ZHANG Y L, et al. A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks[J]. Information Processing & Management, 2021, 58(1): 102437.
[21] SILVA A, LUO L, KARUNASEKERA S, et al. Embracing domain differences in fake news: cross-domain fake news detection using multi-modal data[J]. arXiv:2102,06314, 2021.
[22] XUE J X, WANG Y B, TIAN Y C, et al. Detecting fake news by exploring the consistency of multimodal data[J]. Information Processing & Management, 2021, 58(5): 102610.
[23] NAN Q, CAO J, ZHU Y C, et al. MDFEND: multi-domain fake news detection[C]//Proceedings of the 30th ACM International Conference on Information & Knowledge Management. New York: ACM, 2021: 3343-3347.
[24] XIANG N. Deep learning-based fake information detection and influence evaluation[J]. Computational Intelligence and Neuroscience?, 2022, 2022: 8514430.
[25] LIU P, QIAN W, XU D, et al. Multi-modal fake news detection via bridging the gap between modals[J]. Entropy(Basel), 2023, 25(4): 614.
[26] KUMAR A, TAYLOR J W. Feature importance in the age of explainable AI: case study of detecting fake news & misinformation via a multi-modal framework[J]. European Journal of Operational Research, 2024, 317(2): 401-413.
[27] 刘华玲, 陈尚辉, 乔梁, 等. 多模态混合注意力机制的虚假新闻检测研究[J]. 计算机工程与应用, 2023, 59(9): 95-103.
LIU H L, CHEN S H, QIAO L, et al. Multimodal false news detection based on fusion attention mechanism[J]. Computer Engineering and Applications, 2023, 59(9): 95-103.
[28] ZHANG Y, WALLACE B. A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification[J]. arXiv:1510.03820, 2015.
[29] CHEN J P, ZHANG F, LI H, et al. EMPNet: an extract-map-predict neural network architecture for cross-domain recommendation[J]. World Wide Web, 2024, 27(2): 12.
[30] CHEN J P, LI H Y, ZHANG X D, et al. SR-HetGNN: session-based recommendation with heterogeneous graph neural network[J]. Knowledge and Information Systems, 2024, 66(2): 1111-1134.
[31] CHEN J P, CAO Y, ZHANG F, et al. Sequential intention-aware recommender based on user interaction graph[C]//Proceedings of the International Conference on Multimedia Retrieval. New York: ACM, 2022: 118-126.
[32] LIU X Q, LI X Y, CAO Y, et al. Mandari: multi-modal temporal knowledge graph-aware sub-graph embedding for next-POI recommendation[C]//Proceedings of the IEEE International Conference on Multimedia and Expo. Piscataway: IEEE, 2023: 1529-1534.
[33] DEVLIN J, CHANG M W, LEE K, et al. Bert: Pre-training of deep bidirectional transformers for language understanding[J]. arXiv:1810.04805, 2018.
[34] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision [J]. arXiv:2103.00020, 2021.
[35] GANIN Y, LEMPITSKY V. Unsupervised domain adaptation by backpropagation[J]. arXiv:1409.7495, 2014.
[36] BOIDIDOU C, ANDREADOU K, PAPADOPOULOS S, et al. Verifying multimedia use at MediaEval 2015[C]//Proceedings of the MediaEval 2015 workshop, 2015.
[37] JIN Z W, CAO J, JIANG Y G, et al. News credibility evaluation on microblog with a hierarchical propagation model[C]//Proceedings of the IEEE International Conference on Data Mining. Piscataway: IEEE, 2014: 230-239.
[38] AGRAWAL A, LU J S, ANTOL S, et al. VQA: visual question answering[J]. International Journal of Computer Vision, 2017, 123(1): 4-31.
[39] KHATTAR D, GOUD J S, GUPTA M, et al. MVAE: multimodal variational autoencoder for fake news detection[C]//Proceedings of the World Wide Web Conference. New York: ACM, 2019: 2915-2921.
[40] CHEN Y X, LI D S, ZHANG P, et al. Cross-modal ambiguity learning for multimodal fake news detection[C]//Proceedings
of the ACM Web Conference. New York: ACM, 2022: 2897-2905.
[41] JIANG Y, YU X M, WANG Y M, et al. Similarity-aware multimodal prompt learning for fake news detection[J]. Information Sciences, 2023, 647: 119446.
[42] CHEN Z W, HU L M, LI W X, et al. Causal intervention and counterfactual reasoning for multi-modal fake news detection[C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. Stroudsburg: ACL, 2023: 627-638.
[43] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv:1409. 1556, 2014.
[44] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 770-778.