Image-Guided Augmentation Visual Question Answering Model Combined with Contrastive Learning

doi:10.3778/j.issn.1002-8331.2211-0447

Abstract

Abstract: Aiming at two problems of existing attention-based encoder-decoder visual question answering (VQA) models, image-guided augmentation VQA model combined with contrastive learning (IGA-CL) is proposed. One of these two problems is that single-type image feature contains incomplete visual information, another is that existing models rely overly on question guidance. To solve the first problem, the dual-feature visual decoder (DFVD) is proposed. It is based on the Transformer language encoder. After the single image feature is extended into two types：region and grid, visual information is refined through constructing complementary spatial relations based on the relative positions of different type features. To solve the second problem, the vision-guided language decoder (VGLD) is proposed. It twice matches the two decoded image features with the question features. In which, the parallel gated guided-attention (PGGA) is designed to correct adaptively the guiding proportions of different image features to the question. To obtain more similar mutual information, the contrastive learning loss function is introduced during the training process. It can compare the similarity of different modal features in the hidden space during model reasoning. The proposed model achieves 73.82%, 72.49% and 57.44% overall accuracy on the VQA 2.0, COCO-QA and GQA, respectively, which is 2.92 percentage points, 4.41 percentage points and 0.8 percentage points better than MCAN model. Extensive ablation experiments and visualization analysis demonstrate the effectiveness of the proposed model. Experimental results show that the proposed model can obtain more relevant language-vision information and has stronger generalization ability for different types of question samples.

Key words: visual question answering, attention mechanism, relative position, gated mechanism, contrastive learning

摘要： 针对现有的注意力编解码视觉问答模型存在两个问题：单一形态图像特征包含视觉信息不完整，以及对问题指导过度依赖，提出结合对比学习的图像指导增强视觉问答模型。所提模型包含一种双特征视觉解码器，它基于Transformer语言编码器实现，将单一的图像特征扩展为区域和网格两种形态，根据不同形态特征的相对位置构建互补的空间关系，以解决第一问题。所提模型包含一种视觉引导的语言解码器，将视觉解码的两种图像特征与问题特征二次匹配，通过平行门控引导注意力，自适应地修正不同视觉信息对问题的引导比例，以解决第二问题。所提模型，在训练过程中，引入对比学习损失函数，通过对比模型推理时不同模态特征在隐空间内的相似度，获取更相近的互信息。所提模型，在VQA 2.0、COCO-QA和GQA数据集上分别取得73.82%、72.49%和57.44%的总体准确率，较MCAN模型分别提高2.92个百分点、4.41个百分点和0.8个百分点。大量消融实验和可视化分析证明了模型的有效性。实验结果表明，所提模型能够获取更相关的语言-视觉信息，并且对不同类型的问题样本具有更强的泛化能力。

关键词: 视觉问答, 注意力机制, 相对位置, 门控机制, 对比学习

YANG You, YAO Lu. Image-Guided Augmentation Visual Question Answering Model Combined with Contrastive Learning[J]. Computer Engineering and Applications, 2024, 60(7): 157-166.

杨有, 姚露. 结合对比学习的图像指导增强视觉问答模型[J]. 计算机工程与应用, 2024, 60(7): 157-166.

References

[1] 包希港, 周春来, 肖克晶, 等. 视觉问答研究综述[J]. 软件学报, 2021, 32(8): 2522-2544.
BAO X G, ZHOU C L, XIAO K J, et al. Survey on visual question answering[J]. Journal of Software, 2021, 32(8): 2522-2544.
[2] 王瑞平, 吴士泓, 张美航, 等. 视觉问答语言处理方法综述[J]. 计算机工程与应用, 2022, 58(17): 50-60.
WANG R P, WU S H, ZHANG M H, et al. Review of language processing methods for visual question answering[J]. Computer Engineering and Applications, 2022, 58(17): 50-60.
[3] JIANG H, MISRA I, ROHRBACH M, et al. In defense of grid features for visual question answering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 10267-10276.
[4] ANDERSON P, HE X, BUEHLER C, et al. Bottom-up and top-down attention for image captioning and visual question answering[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 6077-6086.
[5] YU Z, YU J, CUI Y, et al. Deep modular co-attention networks for visual question answering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 6281-6290.
[6] VASWANI A, SHAZEER N, PARMER N, et al. Attention is all you need[C]//Proceeding of the 31st International Conference on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc, 2017: 6000-6010.
[7] YU Z, CUI Y, YU J, et al. Deep multimodal neural architecture search[C]//Proceedings of the 28th ACM International Conference on Multimedia, 2020: 3743-3752.
[8] RAHMAN T, CHOU S H, SIGAL L, et al. An improved attention for visual question answering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 1653-1662.
[9] YANG X, GAO C, ZHANG H, et al. Auto-parsing network for image captioning and visual question answering [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 2197-2207.
[10] ZHOU Y, REN T, ZHU C, et al. TRAR: routing the attention spans in transformer for visual question answering[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 2074-2084.
[11] LIU Y, WEI W, PENG D, et al. Depth-aware and semantic guided relational attention network for visual question answering[J]. IEEE Transactions on Multimedia, 2022, 25: 5344-5357.
[12] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of NAACL-HLT, 2019: 4171-4186.
[13] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
[14] ZHANG X, SUN X, LUO Y, et al. RSTNet: captioning with adaptive attention on visual and non-visual words[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 15465-15474.
[15] HU H, GU J, ZHANG Z, et al. Relation networks for object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 3588-3597.
[16] LUO Y, JI J, SUN X, et al. Dual-level collaborative transformer for image captioning[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2021: 2286-2293.
[17] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//International Conference on Machine Learning, 2021: 8748-8763.
[18] GOYAL Y, KHOT T, SUMMERS-STAU D, et al. Making the V in VQA matter: elevating the role of image understanding in visual question answering[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 6904-6913.
[19] REN M, KIROS R, ZEMEL R. Exploring models and data for image question answering[C]//Proceeding of the 29th Conference on Neural Information Processing Systems, 2015: 2953-2961.
[20] HUDSON D A, MANNING C D. GQA: a new dataset for real-world visual reasoning and compositional question answering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 6700-6709.
[21] PENG L, YANG Y, ZHANG X, et al. Answer again: improving VQA with cascaded-answering model[J]. IEEE Transactions on Knowledge and Data Engineering, 2020, 34(4): 1644-1655.
[22] LIU Y, ZHANG X, ZHANG Q, et al. Dual self-attention with co-attention networks for visual question answering[J]. Pattern Recognition, 2021, 117: 107956.
[23] LIU Y, ZHANG X, HUANG F, et al. Visual question answering via combining inferential attention and semantic space mapping[J]. Knowledge-Based Systems, 2020, 207: 106339.
[24] LIU Y, ZHANG X, ZHAO Z, et al. ALSA: adversarial learning of supervised attentions for visual question answering[J]. IEEE Transactions on Cybernetics, 2020, 52(6): 4520-4533.
[25] PENG L, YANG Y, WANG Z, et al. MRA-Net: improving VQA via multi-modal relation attention network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(1): 318-329.
[26] MAO A, YANG Z, LIN K, et al. Positional attention guided transformer-like architecture for visual question answering[J]. IEEE Transactions on Multimedia, 2022, 25: 6997-7009.
[27] MIAO Y, CHENG W, HE S, et al. Research on visual question answering based on GAT relational reasoning[J]. Neural Processing Letters, 2022, 54(2): 1435-1448.
[28] ZHANG W, YU J, ZHAO W, et al. DMRFNet: deep multimodal reasoning and fusion for visual question answering and explanation generation[J]. Information Fusion, 2021, 72: 70-79.
[29] QIN B, HU H, ZHUANG Y. Deep residual weight-sharing attention network with low-rank attention for visual question answering[J]. IEEE Transactions on Multimedia, 2022, 25: 4282-4295.