Computer Engineering and Applications ›› 2023, Vol. 59 ›› Issue (9): 95-103.DOI: 10.3778/j.issn.1002-8331.2202-0204

• Pattern Recognition and Artificial Intelligence • Previous Articles     Next Articles

Multimodal False News Detection Based on Fusion Attention Mechanism

LIU Hualing, CHEN Shanghui, QIAO Liang, LIU Yaxin   

  1. School of Statistics and Information, Shanghai University of International Business and Economics, Shanghai 201620, China
  • Online:2023-05-01 Published:2023-05-01



  1. 上海对外经贸大学 统计与信息学院,上海 201620

Abstract: Exploring efficient modal representation and multimodal information interaction methods has always been a hot topic in the field of multimodal information detection, for which a new fake news detection technology(MAM) is proposed. The MAM method uses a self-attention mechanism combined with position coding and a pre-trained convolutional neural network to extract text and image features respectively. The introduction of a mixed-attention mechanism module for text and image feature interaction, which uses hierarchical feature processing methods to reduce redundant information generated during multimodal interactions. A two-way feature fusion method is used to ensure the integrity of the training information. The multimodal features are weighted and fed into the fully connected network for true and false news classification. The comparative experimental results show that compared with the existing multimodal reference model, the method is almost improved by about 3 percentage points on each classification index, and the visualization experiment finds that the multimodal features obtained by the mixed attention mechanism have stronger generalization ability.

Key words: false new detection, multimodal analysis, attention mechanism, feature fusion

摘要: 探索高效的模态表示和多模态信息交互方法一直是多模态虚假新闻检测领域的热门话题,提出了一项新的虚假新闻检测技术(MAM)。MAM方法使用结合位置编码的自注意力机制和预训练的卷积神经网络分别提取文本和图像特征;引入混合注意力机制模块进行文本与图像特征交互,该模块使用了层级特征处理方法来减少多模态交互时产生的冗余信息,又使用了双向的特征融合手段保证训练信息的完整性;加权融合多模态特征并将其输入全连接网络中进行真假新闻分类。对比实验结果表明:相比现有的多模态基准模型,该方法几乎在各个分类指标上都提高3个百分点左右,此外,可视化实验发现混合注意力机制获得的多模态特征具有更强的泛化能力。

关键词: 虚假新闻检测, 多模态分析, 注意力机制, 特征融合