Multimodal False News Detection Based on Fusion Attention Mechanism

doi:10.3778/j.issn.1002-8331.2202-0204

Abstract

Abstract: Exploring efficient modal representation and multimodal information interaction methods has always been a hot topic in the field of multimodal information detection, for which a new fake news detection technology（MAM） is proposed. The MAM method uses a self-attention mechanism combined with position coding and a pre-trained convolutional neural network to extract text and image features respectively. The introduction of a mixed-attention mechanism module for text and image feature interaction, which uses hierarchical feature processing methods to reduce redundant information generated during multimodal interactions. A two-way feature fusion method is used to ensure the integrity of the training information. The multimodal features are weighted and fed into the fully connected network for true and false news classification. The comparative experimental results show that compared with the existing multimodal reference model, the method is almost improved by about 3 percentage points on each classification index, and the visualization experiment finds that the multimodal features obtained by the mixed attention mechanism have stronger generalization ability.

Key words: false new detection, multimodal analysis, attention mechanism, feature fusion

摘要： 探索高效的模态表示和多模态信息交互方法一直是多模态虚假新闻检测领域的热门话题，提出了一项新的虚假新闻检测技术（MAM）。MAM方法使用结合位置编码的自注意力机制和预训练的卷积神经网络分别提取文本和图像特征；引入混合注意力机制模块进行文本与图像特征交互，该模块使用了层级特征处理方法来减少多模态交互时产生的冗余信息，又使用了双向的特征融合手段保证训练信息的完整性；加权融合多模态特征并将其输入全连接网络中进行真假新闻分类。对比实验结果表明：相比现有的多模态基准模型，该方法几乎在各个分类指标上都提高3个百分点左右，此外，可视化实验发现混合注意力机制获得的多模态特征具有更强的泛化能力。

关键词: 虚假新闻检测, 多模态分析, 注意力机制, 特征融合

LIU Hualing, CHEN Shanghui, QIAO Liang, LIU Yaxin. Multimodal False News Detection Based on Fusion Attention Mechanism[J]. Computer Engineering and Applications, 2023, 59(9): 95-103.

刘华玲, 陈尚辉, 乔梁, 刘雅欣. 多模态混合注意力机制的虚假新闻检测研究[J]. 计算机工程与应用, 2023, 59(9): 95-103.

References

[1] NGADIRON S，ABD AZIZ A，MOHAMED S S.The spread of Covid-19 fake news on social media and its impact among Malaysians[J].International Journal of Law，Government and Communication，2021，6（22）：253-260.
[2] WU K，YANG S，ZHU K Q.False rumors detection on sina weibo by propagation structures[C]//2015 IEEE 31st International Conference on Data Engineering，2015：651-662.
[3] YANG F，LIU Y，YU X，et al.Automatic detection of rumor on sina weibo[C]//Proceedings of the ACM SIGKDD Workshop on Mining Data Semantics，2012：1-7.
[4] ALAM F，CRESCI S，CHAKRABORTY T，et al.A survey on multimodal disinformation detection[J].arXiv：2103. 12541，2021.
[5] SINGHAL S，SHAH R R，CHAKRABORTY T，et al.Spotfake：a multi-modal framework for fake news detection[C]//2019 IEEE Fifth International Conference on Multimedia Big Data（BigMM），2019：39-47.
[6] KHATTAR D，GOUD J S，GUPTA M，et al.Mvae：multimodal variational autoencoder for fake news detection[C]//The World Wide Web Conference，2019：2915-2921.
[7] BIRADAR S，SAUMYA S，CHAUHAN A.Combating the infodemic：COVID-19 induced fake news recognition in social media networks[J].Complex & Intelligent Systems，2022：1-13.
[8] HAMID A，SHIEKH N，SAID N，et al.Fake news detection in social media using graph neural networks and NLP techniques：a COVID-19 use-case[J].arXiv：2012.07517，2020.
[9] SONG C，NING N，ZHANG Y，et al.A multimodal fake news detection model based on crossmodal attention residual and multichannel convolutional neural networks[J].Information Processing & Management，2021，58（1）：102437.
[10] QIAN S，WANG J，HU J，et al.Hierarchical multi-modal contextual attention network for fake news detection[C]//Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval，2021：153-162.
[11] 高玉君，梁刚，蒋方婷，等.社会网络谣言检测综述[J].电子学报，2020，48（7）：1421-1435.
GAO Y J，LIANG G，JIANG F T，et al.Social network rumor detection：a survey[J].Acta Electronica Sinica，2020，48（7）：1421-1435.
[12] OSHIKAWA R，QIAN J，WANG W Y.A survey on natural language processing for fake news detection[J].arXiv：1811.00770，2018.
[13] WANG Y，HAN H，DING Y，et al.Learning contextual features with multi-head self-attention for fake news detection[C]//International Conference on Cognitive Computing.Cham：Springer，2019：132-142.
[14] BENAMIRA A，DEVILLERS B，LESOT E，et al.Semi-supervised learning and graph neural networks for fake news detection[C]//2019 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining（ASONAM），2019：568-569.
[15] GUPTA M，ZHAO P，HAN J.Evaluating event credibility on twitter[C]//Proceedings of the 2012 SIAM International Conference on Data Mining，2012：153-164.
[16] TIAN D P.A review on image feature extraction and representation techniques[J].International Journal of Multimedia and Ubiquitous Engineering，2013，8（4）：385-396.
[17] SHU K，SLIVA A，WANG S，et al.Fake news detection on social media：a data mining perspective[J].ACM SIGKDD Explorations Newsletter，2017，19（1）：22-36.
[18] JIN Z，CAO J，GUO H，et al.Multimodal fusion with recurrent neural networks for rumor detection on microblogs[C]//Proceedings of the 25th ACM International Conference on Multimedia，2017：795-816.
[19] WANG Y，MA F，JIN Z，et al.Eann：event adversarial neural networks for multi-modal fake news detection[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining，2018：849-857.
[20] QIAN S，HU J，FANG Q，et al.Knowledge-aware multi-modal adaptive graph convolutional networks for fake news detection[J].ACM Transactions on Multimedia Computing，Communications，and Applications（TOMM），2021，17（3）：1-23.
[21] MAYANK M，SHARMA S，SHARMA R.DEAP-FAKED：knowledge graph based approach for fake news detection[J].arXiv：2107.10648，2021.
[22] ZHANG Q，FU J，LIU X，et al.Adaptive co-attention network for named entity recognition in tweets[C]//Thirty-Second AAAI Conference on Artificial Intelligence，2018.
[23] 范涛，吴鹏，王昊，等.基于多模态联合注意力机制的网民情感分析研究[J].情报学报，2021，40（6）：656-665.
FAN T，WU P，WANG H，et al.Sentiment analysis of online users based on multimodal co-attention[J].Journal of the China Society for Secientific and Technical Information，2021，40（6）：656-665.
[24] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems，2017：5998-6008.
[25] CAO J，GUO J，LI X，et al.Automatic rumor detection on microblogs：a survey[J].arXiv：1807.03505，2018.