Multi-Knowledge Base Common Sense Question Answering Model Based on Local Feature Fusion

doi:10.3778/j.issn.1002-8331.2303-0080

Abstract

Abstract: The input and feature combination of the current commonsense reasoning model based on multi-knowledge base fusion is too simple, resulting in the loss of some important information related to questions and answers, which limits the effect of the commonsense reasoning model integrating external knowledge. In addition, during the commonsense question and answer task, the problem of vector anisotropy in the output of the pre-training language model and the answer representation has not been solved. These problems are the factors that lead to the poor reasoning performance of commonsense question answering. To solve the above problems, this paper proposes a multi-knowledge base commonsense question answering model based on local feature fusion, which improves the fusion of external knowledge bases and question-answer texts. The model integrates the local question and answer features into the global features of the pre-trained language model to enrich the feature information of the model, and combines the features of multiple dimensions in the prediction layer for prediction. The model for the questions and answers to be matched. Sentence representations are whitened and then the matching task is performed. Through the whitening operation, the model enhances the isotropy of the sentence representation and improves the representation ability of the sentence vector. This paper also explores the effect of different pre-trained encoders (such as, ALBERT, ELECTRA) on the model to strengthen knowledge. The feature extraction ability of text is strengtened, and the stability of the model is proved. The experimental results show that under the same BERT-base encoder experiment, the accuracy of the model reaches 78.6%, which is 3.5?percentage points higher than the baseline model. In the experiment of ELECTRA-base encoder, the accuracy reaches 80.1%.

Key words: common sense question and answering, knowledge base fusion, local feature fusion prediction, whitening of sentence representations

摘要： 当前的多知识库融合常识推理模型的输入和特征组合的方式过于简单，导致模型丢失了一些与问题和答案相关的重要信息，限制了融合外部知识的常识推理模型的效果。另外，在进行常识问答的任务时，预训练语言模型输出的问题和答案表示存在的向量各向异性问题没有得到解决。这些问题都是导致常识问答推理性能不够高的因素。针对以上问题，提出了一种基于局部特征融合的多知识库常识问答模型，改进外部知识库和问答文本的融合方式。模型将局部的问题和答案特征融入预训练语言模型全局特征，以丰富模型的特征信息，并在预测层结合了多种维度的特征进行预测；模型对于待匹配的问题和答案句子表示进行了白化处理，然后执行匹配任务。通过白化操作，模型增强了句子表示的各向同性，提升了句子向量的表征能力；还探索了不同预训练编码器（如：ALBERT、ELECTRA）在模型上的效果，以加强对知识文本的特征抽取能力，并证明了模型的稳定性。实验结果证明，在相同BERT-base编码器的实验下，模型的准确率达到78.6%，相较于基线模型，准确率提升了3.5个百分点；在ELECTRA-base编码器的实验下，模型的准确率达到80.1%。

关键词: 常识问答, 知识库融合, 局部特征融合预测, 向量白化

TIAN Yuqing, WANG Chunmei, YUAN Feiniu. Multi-Knowledge Base Common Sense Question Answering Model Based on Local Feature Fusion[J]. Computer Engineering and Applications, 2024, 60(12): 129-135.

田雨晴, 汪春梅, 袁非牛. 融合局部特征的多知识库常识问答模型[J]. 计算机工程与应用, 2024, 60(12): 129-135.

References

[1] RAJPURKAR P, JIA R, LIANG P. Know what you don’t know: unanswerable questions for SQuAD[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2018: 784-789.
[2] ZELLERS R, BISK Y, SCHWARTZ R, et al. SWAG: a large-scale adversarial dataset for grounded commonsense inference[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018: 93-104.
[3] GREFF K, SRIVASTAVA R K, KOUTNíK J, et al. LSTM: a search space odyssey[J]. IEEE Transactions on Neural Networks & Learning Systems, 2016, 28(10): 2222-2232.
[4] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 6000-6010.
[5] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2018: 4171-4186.
[6] MIKOLOV T, CORRADO G, KAI C, et al. Efficient estimation of word representations in vector space[C]//International Conference on Learning Representations, 2013.
[7] PENNINGTON J, SOCHER R, MANNING C D. Glove: global vectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014: 1532-1543.
[8] TALMOR A, HERZIG J, LOURIE N, et al. CommonsenseQA: a question answering challenge targeting commonsense knowledge[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019: 4149-4158.
[9] SPEER R, CHIN J, HAVASI C. ConceptNet 5.5: an open multilingual graph of general knowledge[C]//Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, 2017: 4444-4451.
[10] XU Y, ZHU C, WANG S, et al. Human parity on commonsenseQA: augmenting self-attention with external attention[C]//Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence Main Track, 2021: 2762-2768.
[11] CLARK K, LUONG M T, LE Q V, et al. ELECTRA: pre-training text encoders as discriminators rather than generators[C]//2020 International Conference on Learning Representations, 2020.
[12] LIN B Y, CHEN X, CHEN J, et al. KagNet: knowledge-aware graph networks for commonsense reasoning[C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019: 2829-2839.
[13] YASUNAGA M, REN H, BOSSELUT A, et al. QA-GNN: reasoning with language models and knowledge graphs for question answering[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021: 535-546.
[14] CUI W, XIAO Y, WANG H, et al. KBQA: learning question answering over QA corpora and knowledge bases[J]. Proceedings of the VLDB Endowment, 2017, 10(5): 565-576.
[15] WU Y, LIU X, FENG Y, et al. Relation-aware entity alignment for heterogeneous knowledge graphs[C]//Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, 2019: 5278-5284.
[16] YASUNAGA M, BOSSELUT A, REN H, et al. Deep bidirectional language?knowledge graph pretraining[C]//36th Conference on Neural Information Processing Systems (NeurIPS 2022), 2022.
[17] LV S, GUO D, XU J, et al. Graph-based reasoning over heterogeneous external knowledge for commonsense question answering[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2020: 8449-8456.
[18] RAJANI N F, MCCANN B, XIONG C, et al. Explain Yourself! Leveraging language models for commonsense reasoning[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019: 4932-4942.
[19] ZELIKMAN E, WU Y, MU J, et al. Star: bootstrapping reasoning with reasoning[C]//Advances in Neural Information Processing Systems, 2022: 15476-15488.
[20] WU S, IRSOY O, LU S, et al. Bloomberggpt: a large language model for finance[J]. arXiv:2303.17564, 2023.
[21] LAN Z, CHEN M, GOODMAN S, et al. ALBERT: a lite bert for self-supervised learning of language representations[C]//2020 International Conference on Learning Representations, 2019.
[22] ZESCH T, MüLLER C, GUREVYCH I. Extracting lexical semantic knowledge from wikipedia and wiktionary[C]//Proceedings of the International Conference on Language Resources and Evaluation (LREC 2008), Marrakech, Morocco, 2008: 1646-1652.
[23] SU J, CAO J, LIU W, et al. Whitening sentence representations for better semantics and faster retrieval[J]. arXiv:2103.15316, 2021.