Abstract: In multimodal sentiment analysis, the modality that plays a dominant role in sentiment determination is dynamic. Usually, traditional multimodal sentiment analysis methods regard text modal as a dominant modal, but ignore the change in dominant modal at different moments due to the differences between modalities. Aiming at selecting dominant modal dynamically in each moment, this paper proposes a dynamic dominant fusion multimodal sentiment analysis method based on autoencoder. The method firstly encodes single modalities and obtains multimodal fusion features. And an autoencoder is applied to map them into a shared space. In the space, the dominant modality is selected by correlation between unimodal and fusion modal. Finally, the dominant multimodal information is used to guide multimodal fusion to obtain the multimodal robustness representation. The extensive experiments on the multimodal sentiment analysis benchmark dataset CMU-MOSI demonstrate the effectiveness of the proposed method, which outperform most of the existing state-of-the-art multimodal sentiment analysis methods.

Key words: multimodal sentiment analysis, dynamic complementarity, dominant modality, autoencoder

摘要： 多模态情感分析过程中，对情感判定起主导作用的模态常常是动态变化的。传统多模态情感分析方法中通常仅以文本为主导模态，而忽略了由于模态之间的差异性造成不同时刻主导模态的变化。针对如何在各个时刻动态选取主导模态的问题，提出一种自编码器动态主导融合的多模态情感分析方法。该方法首先对单模态编码并获得多模态融合特征，再利用自编码器将其表征到共享空间内；在此空间内衡量单模态特征与融合模态特征的相关程度，在各个时刻动态地选取相关程度最大的模态作为该时刻的主导模态；最后，利用主导模态引导多模态信息融合，得到多模态鲁棒性表征。在多模态情感分析基准数据集CMU-MOSI上进行广泛实验，实验结果表明提出方法的有效性，并且优于大多数现有最先进的多模态情感分析方法。

关键词: 多模态情感分析, 动态互补, 主导模态, 自编码器