关键词: 情感分类, 多模态数据, 双向注意力机制, 后期融合

Abstract:

The development of social network provides large amounts of multimodal data for sentiment analysis. Sentiment classification based on multimodal content can exploit the relative information between various modalities, avoiding the incomplete grasp of the overall emotion. Simple fusion methods cannot fully excavate the complementary characteristics of multiple modalities, therefore a Multimodal Bidirectional Attention Hybrid model（MBAH） is proposed. Based on the image and text features extracted from the deep models, another modality information is introduced on modality by bidirectional attention mechanism and the low-level features of this modality and the semantic features of another modality are calculated to learn the association information between the modalities through attention. Then it assembles the high-level features of the two modalities to form a cross-modal shared representation and inputs into the multilayer perceptron to obtain the classification result. In addition, the MBAH model combines with the image-textunimodal self-attention models search for the optimal decision weights through late fusion to form the final decision. Experimental result shows that the MBAH model outperforms other methods on sentiment classification.

Key words: sentiment classification, multimodal data, bidirectional attention mechanism, late fusion