Cross-Modal Representation Learning Under Multi-Negatives Contrastive Mechanism

doi:10.3778/j.issn.1002-8331.2102-0060

Abstract

Abstract: In order to obtain more distinctive cross-modal representations effectively, a cross-modal representation learning method based on the multi-negatives contrastive mechanism—supervised contrastive cross-modal representation learning（SCCMRL） is proposed, and it is applies to the modalities of vision and audio. SCCMRL extracts vision and audio features through vision encoder and audio encoder which uses supervised contrastive loss to compare sample with its multiple negatives. As a result, the audio-visual features that belong to the same category are closer, and the audio-visual features that belong to different categories are more distant. Furthermore, this method also introduces center loss and label loss to ensure the modality consistency and semantic discrimination between cross-modal representations. To verify the effectiveness of the SCCMRL method, this paper constructs a corresponding cross-modal retrieval system, which conducts cross-modal retrieval experiments based on the Sub_URMP and XmediaNet datasets. The experimental results show that the SCCMRL method has achieved a higher mAP value than the current cross-modal retrieval methods that are used commonly. It also verifies the feasibility of applying the multi-negatives contrastive mechanism in cross-modal representation learning.

Key words: cross-modal representation learning, multimodal feature fusion, multi-negatives contrastive mechanism, supervised contrastive loss, cross-modal retrieval

摘要： 为了有效地获取到更有区别性的跨模态表示，提出了一种基于多负例对比机制的跨模态表示学习方法——监督对比的跨模态表示学习（supervised contrastive cross-modal representation learning，SCCMRL），并将其应用于视觉模态和听觉模态上。SCCMRL分别通过视觉编码器和音频编码器提取得到视听觉特征，利用监督对比损失让样本数据与其多个负例进行对比，使得相同类别的视听觉特征距离更近，不同类别的视听觉特征距离更远。此外，该方法还引入了中心损失和标签损失来进一步保证跨模态表示间的模态一致性和语义区分性。为了验证SCCMRL方法的有效性，基于SCCMRL方法构建了相应的跨模态检索系统，并结合Sub_URMP和XmediaNet数据集进行了跨模态检索实验。实验结果表明，SCCMRL方法相较于当前常用的跨模态检索方法取得了更高的mAP值，同时验证了多负例对比机制下的跨模态表示学习具有可行性。

关键词: 跨模态表示学习, 多模态特征融合, 多负例对比机制, 监督对比损失, 跨模态检索

DING Kaixuan, CHEN Yanxiang, ZHAO Pengcheng, ZHU Yupeng, SHENG Zhentao. Cross-Modal Representation Learning Under Multi-Negatives Contrastive Mechanism[J]. Computer Engineering and Applications, 2022, 58(19): 184-192.

丁凯旋, 陈雁翔, 赵鹏铖, 朱玉鹏, 盛振涛. 多负例对比机制下的跨模态表示学习[J]. 计算机工程与应用, 2022, 58(19): 184-192.

References

[1] BALTRU?AITIS T，AHUJA C AND MORENCY L P.Multimodal machine learning：A survey and taxonomy[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2018，41（2）：423-443.
[2] HONG R，HU Z，WANG R，et al.Multi-view object retrieval via multi-scale topic models[J].IEEE Transactions on Image Processing，2016，25（12）：5814-5827.
[3] FENG F，WANG X，LI R.Cross-modal retrieval with correspondence autoencoder[C]//Proceedings of the 22nd ACM International Conference on Multimedia，November，2014：7-16.
[4] RASIWASIA N，COSTA PEREIRA J，COVIELLO E，et al.A new approach to cross-modal multimedia retrieval[C]//Proceedings of the 18th ACM International Conference on Multimedia，October 25-29，2010：251-260.
[5] VINYALS O，TOSHEV A，BENGIO S，et al.Show and tell：A neural image caption generator[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：3156-3164.
[6] CHEN L，ZHANG H，XIAO J，et al.SCA-CNN：Spatial and channel-wise attention in convolutional networks for image captioning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：6298-6306.
[7] 陈龙杰，张钰，张玉梅，等.基于多注意力多尺度特征融合的图像描述生成算法[J].计算机应用，2019，39（2）：354-359.
CHEN L J，ZHANG Y，ZHANG Y M，et al.Image ption generation algorithm based on multi-attention and multi-scale feature fusion[J].Journal of Computer Applications，2019，39（2）：354-359.
[8] YANG Z，HE X，GAO J，et al.Stacked attention networks for image question answering[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：21-29.
[9] 闫茹玉，刘学亮.结合自底向上注意力机制和记忆网络的视觉问答模型[J].中国图象图形学报，2020，25（5）：993-1006.
YAN R Y，LIU X L.Visual question answering model combining bottom-up attention mechanism and memory network[J].Journal of Image and Graphics，2020，25（5）：993-1006.
[10] HARDOON D R，SZEDMAK S，SHAWE-TAYLOR J.Canonical correlation analysis：An overview with application to learning methods[J].Neural Computation，2004，16（12）：2639-2664.
[11] AKAHO S.A kernel method for canonical correlation analysis[EB/OL].（2001-10-03）[2021-01-15].https：//arxiv/ abs/cs/0609071.
[12] ANDREW G，ARORA R，BILMES J，et al.Deep canonical correlation analysis[C]//Proceedings of the 30th International Conference on International Conference on Machine Learning，June，2013：1247-1255.
[13] SHARMA A，KUMAR A，DAUME H，et al.Generalized multiview analysis：A discriminative latent space[C]//Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition，2012：2160-2167.
[14] WANG K，HE R，WANG W，et al.Learning coupled feature spaces for cross-modal matching[C]//Proceedings of 2013 IEEE International Conference on Computer Vision，2013：2088-2095.
[15] WANG B，YANG Y，XU X，et al.Adversarial cross-modal retrieval[C]//Proceedings of the 25th ACM International Conference on Multimedia，October，2017：154-162.
[16] ZHEN L，HU P，WANG X，et al.Deep supervised cross-modal retrieval[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），2019：10386-10395.
[17] PENG Y X，HUANG X，QI J W.Cross-media shared representation by hierarchical learning with multiple deep networks[C]//Proceedings of the 25th International Joint Conference on Artificial Intelligence，July，2016：3846-3853.
[18] PENG Y，QI J，HUANG X，et al.CCL：Cross-modal correlation learning with multi grained fusion by hierarchical network[J].IEEE Transactions on Multimedia，2017，20（2）：405-420.
[19] SALVADOR A，HYNES N，AYTAR Y，et al.Learning cross-modal embeddings for cooking recipes and food images[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition（CVPR），2017：3068-3076.
[20] SURíS D，DUARTE A，SALVADOR A，et al.Cross-modal embeddings for video and audio retrieval[C]//Proceedings of ECCV 2018 Workshops，2018：711-716.
[21] ZENG D，YU Y，OYAMA K.Deep triplet neural networks with cluster-CCA for audio-visual cross-modal retrieval[J].ACM Transactions on Multimedia Computing，Communications，and Applications（TOMM），2020，16（3）：1-23.
[22] KHOSLA P，TETERWAK P，WANG C，et al.Supervised contrastive learning[EB/OL].（2020-12-10）[2021-01-15].https：//arxiv.org/abs/2004.11362v1.
[23] HADSELL R，CHOPRA S，LECUN Y.Dimensionality reduction by learning an invariant mapping[C]//Proceedings of 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2006：1735-1742.
[24] SCHROFF F，KALENICHENKO?D，PHILBIN?J.Facenet：A unified embedding for face recognition and clustering[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition，2015：815-823.
[25] HE K M.Momentum contrast for unsupervised visual representation learning[EB/OL].（2019-11-13）[2021-01-15].https：arxiv.org/abs/1911.05722.
[26] DAI?B，DAHUA?L.Contrastive learning for image captioning[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems，December，2017：898-907.
[27] OORD A.Representation learning with contrastive predictive coding[EB/OL].（2019-01-22）[2021-01-15].https：//arxiv.org/abs/1807.03748.
[28] HONG R，LI L，CAI J，et al.Coherent semantic-visual indexing for large-scale image retrieval in the cloud[J].IEEE Transactions on Image Processing，2017，26（9）：428-438.
[29] LIANG J，HE R，SUN Z，et al.Group-invariant cross-modal subspace learning[C]//Proceedings of the 25th International Joint Conference on Artificial Intelligence，July，2016：1739-1745.
[30] NGIAM J，KHOSLA A，KIM M，et al.Multimodal deep learning[C]//Proceedings of the 28th International Conference on Machine Learning，2011：689-696.
[31] WANG C，YANG H，MEINEL C.Deep semantic mapping for cross-modal retrieval[C]//Proceedings of the 2015 IEEE 27th International Conference on Tools with Artificial Intelligence，2015：234-241.
[32] KUMAR S，UDUPA R.Learning hash functions for cross-view similarity search[C]//Proceedings of the 22nd International Joint Conference on Artificial Intelligence，July，2011：1360-1365.
[33] DING G，GUO Y，ZHOU J.Collective matrix factorization hashing for multimodal data[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition，June，2014：2083-2090.
[34] WANG D，CUI P，OU M，et al.Learning compact hash codes for multimodal representations using orthogonal deep structure[J].IEEE Transactions on Multimedia，2015，17（9）：1404-1416.
[35] ARANDJELOVIC R，ZISSERMAN A.Look，listen and learn[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision，2017：609-617.
[36] HU P，ZHEN L，PENG D，et al.Scalable deep multimodal learning for cross-modal retrieval[C]//Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval，July，2019：635-644.