Cross-Modal Modulating for Multimodal Sentiment Analysis

doi:10.3778/j.issn.1002-8331.2201-0406

Abstract

Abstract: How to effectively represent modalities and efficiently integrate information between modalities has always been a hot issue in the field of multimodal sentiment analysis（MSA）. Most of the existing research is based on the Transformer, and the self-attention module is improved to achieve the effect of cross-modal fusion. However, the fusion method based on the Transformer often ignores the importance of different modalities, and the Transformer cannot effectively capture the temporal features. In response to the above problems, a cross-modal modulating and multimodal gating module network is proposed, which uses the LSTM and the BERT as the representation sub-networks of visual, acoustic and text modalities respectively. The improved Transformer cross-modal modulation module is used to effectively fuse different modal information. A modal gating network is designed to simulate the synthetic judgment process of information from different modes. Comparative experiments are carried out using MOSI and MOSEI datasets, and the results show that the proposed method can effectively improve the accuracy of sentiment classification.

Key words: multimodal sentiment analysis（MSA）, Transformer model, cross-modal modulating, multimodal gating network

摘要： 如何对模态进行有效表示和对模态间信息进行高效融合，一直是多模态情感分析领域的一个热点问题。已有研究大都以Transformer为基础，对其中自注意力模块进行改进以达到跨模态融合的效果。但基于Transformer的融合方式往往忽略了不同模态之间的重要程度，同时Transformer无法有效地捕捉到时间特征。为此，提出了基于跨模态调制及模态门控网络模型。该模型利用LSTM网络和BERT分别作为视觉、听觉和文本模态的表示子网络；利用改进的Transformer模型的跨模态调制模块对不同的模态信息进行有效的融合；设计了模态门控网络，模拟人类对来自不同模态的信息进行综合的判断。利用MOSI、MOSEI数据集进行了对比实验，结果表明所提出的方法有效地提高了情感分类的准确度。

关键词: 多模态情感分析（MSA）, Transformer模型, 跨模态调制, 模态门控网络

CHENG Zichen, LI Yan, GE Jiangwei, JIU Mengfei, ZHANG Jingwei. Cross-Modal Modulating for Multimodal Sentiment Analysis[J]. Computer Engineering and Applications, 2023, 59(10): 171-179.

程子晨, 李彦, 葛江炜, 纠梦菲, 张敬伟. 利用跨模态调制的多模态情感分析[J]. 计算机工程与应用, 2023, 59(10): 171-179.

References

[1] HAN W，CHEN H，PORIA S.Improving multimodal fusion with hierarchical mutual information maximization for multimodal sentiment analysis[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing，Punta Cana.Stroudsburg：ACL，2021：9180-9192.
[2] BALTRUSAITIS T，AHUJA C，MORENCY L-P.Multimodal machine learning：a survey and taxonomy[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2019，41（2）：423-443.
[3] SIRIWARDHANA S，KALUARACHCHI T，BILLINGHURST M，et al.Multimodal emotion recognition with transformer-based self supervised feature fusion[J].IEEE Access，2020，8：176274-176285.
[4] DEGOTTEX G，KANE J，DRUGMAN T，et al.COVAREP—a collaborative voice analysis repository for speech technologies[C]//2014 IEEE International Conference on Acoustics，Speech and Signal Processing，2014：960-964.
[5] PENNINGTON J，SOCHER R，MANNING C.GloVe：global vectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing，Doha.Stroudsburg：ACL，2014：1532-1543.
[6] ZADEH A，CHEN M，PORIA S，et al.Tensor fusion network for multimodal sentiment analysis[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing，Copenhagen.Stroudsburg：ACL，2017：1103-1114.
[7] LIU Z，SHEN Y，LAKSHMINARASIMHAN V B，et al.Efficient low-rank multimodal fusion with modality-specific factors[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics（Volume 1：Long Papers），Melbourne.Stroudsburg：ACL，2018：2247-2256.
[8] ZADEH A，LIANG P P，MAZUMDER N，et al.Memory fusion network for multi-view sequential learning[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence，2018.
[9] BAHDANAU D，CHO K，BENGIO Y.Neural machine translation by jointly learning to align and translate[J].arXiv：1409.0473，2014.
[10] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems 30：Annual Conference on Neural Information Processing Systems 2017，Dec 4-9，2017：5998-6008.
[11] TSAI Y-H H，BAI S，LIANG P P，et al.Multimodal transformer for unaligned multimodal language sequences[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics，Florence.Stroudsburg：ACL，2019：6558-6569.
[12] PARK S，SHIM H S，CHATTERJEE M，et al.Computational analysis of persuasiveness in social multimedia：a novel dataset and multimodal prediction approach[C]//Proceedings of the 16th International Conference on Multimodal Interaction，Istanbul.New York：ACM，2014：50-57.
[13] 张峰，李希城，董春茹，等.基于深度情感唤醒网络的多模态情感分析与情绪识别[J].控制与决策，2022，37（11）：2984-2992.
ZHANG F，LI X C，DONG C R，et al.Deep emotional arousal network for multimodal sentiment analysis and emotion recognition[J].Control and Decision，2022，37（11）：2984-2992.
[14] ZENG Y，MAI S，HU H.Which is making the contribution：modulating unimodal and cross-modal dynamics for multimodal sentiment analysis[C]//Findings of the Association for Computational Linguistics：EMNLP 2021，Punta Cana.Stroudsburg：ACL，2021：1262-1274.
[15] WANG Y，SHEN Y，LIU Z，et al.Words can shift：dynamically adjusting word representations using nonverbal behaviors[C]//Proceedings of the 33rd AAAI Conference on Artificial Intelligence，2019：7216-7223.
[16] DELBROUCK J-B，TITS N，DUPONT S.Modulated fusion using transformer for linguistic-acoustic emotion recognition[C]//Proceedings of the 1st International Workshop on Natural Language Processing Beyond Text.Stroudsburg：ACL，2020：1-10.
[17] RAHMAN W，HASAN M K，LEE S，et al.Integrating multimodal information in large pretrained transformers[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics，2020：2359-2369.
[18] BAGHER ZADEH A，LIANG P P，PORIA S，et al.Multimodal language analysis in the wild：CMU-MOSEI dataset and interpretable dynamic fusion graph[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics（Volume 1：Long Papers），Melbourne.Stroudsburg：ACL，2018：2236-2246.
[19] HAZARIKA D，ZIMMERMANN R，PORIA S.MISA：modality-invariant and -specific representations for multimodal sentiment analysis[C]//Proceedings of the 28th ACM International Conference on Multimedia，Seattle.New York：ACM：1122-1131.
[20] YU W，XU H，YUAN Z，et al.Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis[C]//Proceedings of the 35th AAAI Conference on Artificial Intelligence，the 33rd Conference on Innovative Applications of Artificial Intelligence，the 11th Symposium on Educational Advances in Artificial Intelligence，Feb 2-9，2021.Menlo Park：AAAI，2021：10790-10797.
[21] W?LLMER M，WENINGER F，KNAUP T，et al.YouTube movie reviews：sentiment analysis in an audio-visual context[J].IEEE Intelligent Systems，2013，28（3）：46-53.
[22] HAN W，CHEN H，GELBUKH A，et al.Bi-bimodal modality fusion for correlation-controlled multimodal sentiment analysis[G]//Proceedings of the 2021 International Conference on Multimodal Interaction.New York：ACM，2021：6-15.
[23] KUMAR A，VEPA J.Gated mechanism for attention based multi modal sentiment analysis[C]//2020 IEEE International Conference on Acoustics，Speech and Signal Processing，2020：4477-4481.
[24] DELBROUCK J-B，TITS N，BROUSMICHE M，et al.A Transformer-based joint-encoding for emotion recognition and sentiment analysis[C]//2nd Grand-Challenge and Workshop on Multimodal Language（Challenge-HML），Seattle.Stroudsburg：ACL，2020：1-7.
[25] 赵宏，傅兆阳，赵凡.基于BERT和层次化Attention的微博情感分析研究[J].计算机工程与应用，2022，58（5）：156-162.
ZHAO H，FU Z Y，ZHAO F.Microblog sentiment analysis based on BERT and hierarchical attention[J].Computer Engineering and Applications，2022，58（5）：156-162.
[26] DEVLIN J，CHANG M-W，LEE K，et al.BERT：pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，Minneapolis.Stroudsburg：ACL.2019：4171-4186.
[27] 胡任远，刘建华，卜冠南，等.融合BERT的多层次语义协同模型情感分析研究[J].计算机工程与应用，2021，57（13）：176-184.
HU R Y，LIU J H，BU G N，et al.Research on sentiment analysis of multi-level semantic collaboration model fused with BERT[J].Computer Engineering and Applications，2021，57（13）：176-184.
[28] YANG K，XU H，GAO K.CM-BERT：cross-modal BERT for text-audio sentiment analysis[C]//Proceedings of the 28th ACM International Conference on Multimedia.New York：ACM，2020：521-528.
[29] ZADEH A，ZELLERS R，PINCUS E，et al.MOSI：multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos[J].arXiv：1606.06259，2016.
[30] YU W，XU H，MENG F，et al.CH-SIMS：a Chinese multimodal sentiment analysis dataset with fine-grained annotation of modality[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.Stroudsburg：ACL，2020：3718-3727.