融合改进图卷积的跨模态检索

doi:10.3778/j.issn.1002-8331.2302-0064

摘要/Abstract

摘要： 针对现有跨模态检索在公共子空间度量时难以充分挖掘模态内局部一致性的问题，提出了一种融合改进图卷积的跨模态检索方法。为了提升各模态内的局部一致性，以单个完整样本为节点构建模态图，充分挖掘特征间的交互信息；为了解决图卷积网络只能做浅层学习的问题，采用在每一层图卷积添加初始残差链接和权重恒等映射的方法来缓解此现象；为了通过高低阶邻居信息共同更新中心节点特征，提出减少邻居节点、增加图卷积网络层数的改进；为了学习高度局部一致且语义一致的公共表征，共享公共表征学习层权重，并联合优化公共子空间中模态内的语义约束和模态间的模态不变约束。实验结果表明，在Wikipedia和Pascal sentence这两个跨模态数据集上，不同检索任务的平均mAP值比11种现有方法分别提升了2.2%~42.1%和3.0%~54.0%。

关键词: 图卷积网络, 跨模态检索, 初始残差连接, 恒等映射, 邻接矩阵

Abstract: Aiming at the problem that existing image text cross-modal retrieval is difficult to fully exploit the local consistency in the mode in the common subspace, a cross-modal retrieval method based on improved graph convolution is proposed. In order to improve the local consistency within each mode, the modal diagram is constructed with a single sample as a node, fully mining the interactive information between features. In order to solve the problem that graph convolution network can only do shallow learning, the method of adding initial residual link and weight identity map in each layer of graph convolution is adopted to alleviate this phenomenon. In order to jointly update the central node features through higher-order and lower-order neighbor information, an improvement is proposed to reduce neighbor nodes and increase the number of layers in graph convolution network. In order to learn highly locally consistent and semantically consistent public representation, it shares the weights of common representation learning layer, and jointly optimizes the semantic constraints within the modes and the modal invariant constraints between modes in the common subspace. The experimental results show that on the two cross-modal data sets of Wikipedia and Pascal sentence, the average mAP values of different retrieval tasks are 2.2%~42.1% and 3.0%~54.0% higher than the 11 existing methods.

Key words: graph convolution network, cross-modal retrieval, initial residual connection, identity mapping, adjacency matrix

张宏图, 化春键, 蒋毅, 俞建峰, 陈莹. 融合改进图卷积的跨模态检索[J]. 计算机工程与应用, 2024, 60(11): 95-104.

ZHANG Hongtu, HUA Chunjian, JIANG Yi, YU Jianfeng, CHEN Ying. Cross-Modal Retrieval with Improved Graph Convolution[J]. Computer Engineering and Applications, 2024, 60(11): 95-104.

参考文献

[1] ZENG C, BAI C, MA Q, et al. Adversarial projection learning based hashing for cross-modal retrieval[J]. Journal of Computer-Aided Design & Computer Graphics, 2021, 33(6): 904-912.
[2] 代瑾, 陈莹. 联合线性判别和图正则的任务导向型跨模态检索[J]. 计算机辅助设计与图形学学报, 2021, 33(1): 106-115.
DAI J, CHEN Y. Joint linear discrimination and graph regularization for task-oriented cross-modal retrieval[J]. Journal of Computer-Aided Design & Computer Graphics, 2021, 33(1): 106-115.
[3] WEI X, ZHANG T Z, LI Y, et al. Multi-modality cross attention network for image and sentence matching[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 10938-10947.
[4] 刘颖, 郭莹莹, 房杰, 等. 深度学习跨模态图文检索研究综述[J]. 计算机科学与探索, 2022, 16(3): 489-511.
LIU Y, GUO Y Y, FANG J, et al. Survey of research on deep learning image-text cross-modal retrieval[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(3): 489-511.
[5] LI M Y, LI Y, HUANG S L, et al. Semantically supervised maximal correlation for cross-modal retrieval[C]//Proceedings of the 27th IEEE International Conference on Image Processing, 2020: 2291-2295.
[6] CHEN N, DUAN Y X, SUN Q F. Literature review of cross-modal retrieval research[J]. Journal of Frontiers of Computer Science and Technology, 2021, 15(8): 1390-1404.
[7] NGIAM J, KHOSLA A, KIM M, et al. Multimodal deep learning[C]//Proceedings of the International Conference on Machine Learning, 2011: 689-696.
[8] FENG F X, WANG X J, LI R F. Cross-modal retrieval with correspondence autoencoder[C]//Proceedings of the ACM International Conference on Multimedia, 2014: 7-16.
[9] WEI Y C, ZHAO Y, LU C, et al. Cross-modal retrieval with CNN visual features: a new baseline[J]. IEEE Transactions on Cybernetics, 2016, 47(2): 449-460.
[10] PENG Y X, HUANG X, QI J W. Cross-media shared representation by hierarchical learning with multiple deep networks[C]//Proceedings of the International Joint Conference on Artificial Intelligence, 2016: 3846-3853.
[11] WANG B K, YANG Y, XU X, et al. Adversarial cross-modal retrieval[C]//Proceedings of the ACM International Conference on Multimedia, 2017: 154-162.
[12] OU W, XUAN R, GOU J, et al. Semantic consistent adversarial cross-modal retrieval exploiting semantic similarity[J]. Multimedia Tools and Applications, 2020, 79(21/22): 14733-14750.
[13] ZHEN L L, HU P, WANG X, et al. Deep supervised cross-modal retrieval[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 10394-10403.
[14] XU B B, CEN K Y, HUANG J J, et al. A Survey on graph convolutional neural network[J]. Chinese Journal of Computers, 2020, 43(5): 755-780.
[15] WU Z H, PAN S R, CHEN F W, et al. A comprehensive survey on graph neural networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 32(1): 4-24.
[16] LI K, ZHANG Y, LI K, et al. Visual semantic reasoning for image-text matching[C]//Proceedings of the IEEE/CVF International Conference on Computer Visions, 2019: 4653-4661.
[17] NORCLIFFE-BROWN W, VAFEIAS E, PARISOT S. Learning conditioned graph structures for interpretable visual question answering[C]//Proceedings of the International Conference on Neural Information Processing Systems, 2018: 8344-8353.
[18] CHEN M, WEI Z, HUANG Z, et al. Simple and deep graph convolutional networks[C]//Proceedings of the International Conference on Machine Learning, 2020: 1725-1735.
[19] WANG W, ARORA R, LIVESCU K, et al. On deep multi-view representation learning[C]//Proceedings of the International Conference on Machine Learning, 2015: 1083-1092.
[20] RASIWASIA N, COSTA-PEREIRA J, COVIELLO E, et al. A new approach to cross-modal multimedia retrieval[C]//Proceedings of the ACM International Conference on Multimedia, 2010: 251-260.
[21] RASHTCHIAN C, YOUNG P, HODOSH M, et al. Collecting image annotations using Amazon’s mechanical turk[C]//Proceedings of the Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk, 2010: 139-147.
[22] KINGMA D P, BA J. ADAM: a method for stochastic optimization[J]. arXiv:1412.6980, 2014.
[23] HOTELLING H. Relations between two sets of variates[J]. Breakthroughs in Statistics: Methodology and Distribution, 1992: 162-190.
[24] RUPNIK J, SHAWE-TAYLOR J. Multi-view canonical correlation analysis[C]//Proceedings of the Conference on Database, Data Warehouses, Data Mining and Big Data, 2010: 1-4.
[25] KAN M, SHAN S, ZHANG H, et al. Multi-view discriminant analysis[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 38(1): 188-194.
[26] ZHAI X, PENG Y, XIAO J. Learning cross-media joint representation with sparse and semisupervised regularization[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(6): 965-978.
[27] PENG Y X, QI J W, YUAN Y X. CM-GANs: cross-modal generative adversarial networks for common representation learning[J]. ACM Transactions on Multimedia Computing Communications and Applications, 2019, 15(1): 1-24.
[28] ANDREW G, ARORA R, BILMES J, et al. Deep canonical correlation analysis[C]//Proceedings of the International Conference on Machine Learning, 2013: 1247-1255.