关键词: 多模态特征融合, 分级特征, 映射与融合, 情感分类, 特征表示

Abstract: Multimodal feature representation serves as the foundation for multimodal tasks. To address the issue of a single-level fusion in existing multimodal feature representation methods, which fails to adequately capture the inter-modal relationships, a novel approach for multimodal hierarchical feature mapping and fusion representation is proposed. This method, built upon the text model RoBERTa and the image model DenseNet, extracts features from intermediate layers of both models spanning from low to high levels. Leveraging the concept of feature reuse, it maps and fuses features at different levels of the text and image modalities, capturing the internal relationships between text and image modalities and effectively integrating features between the two modalities. The hierarchical feature mapping and fusion representation is then fed into a classifier for sentiment classification in the context of multimodal sentiment analysis. A comparative analysis is also conducted between the constructed representation method and baseline representation methods. The experimental results indicate that the proposed representation method surpasses all baseline models in terms of sentiment classification performance on both the Weibo sentiment and MVSA-Multiple datasets. Specifically, it achieves a 0.013?7 increase in F1 score on the Weibo dataset and a 0.022?2 increase on the MVSA-Multiple dataset. Image features enhance sentiment classification accuracy under the single modality of text, but the degree of improvement is closely tied to the fusion strategy. The multimodal hierarchical feature mapping and fusion representation method effectively maps the relationship between text and image features, ultimately improving the effectiveness of sentiment classification in multimodal sentiment analysis.

Key words: multimodal feature fusion, hierarchical feature, map and fusion, sentiment classification, feature representation