可解释性逻辑推理数据集的构建和研究

doi:10.3778/j.issn.1002-8331.2309-0458

摘要/Abstract

摘要： 逻辑推理能力对于机器和人类理解自然语言具有重要的意义。逻辑推理问题的解释是对逻辑推理过程的阐述和说明，但在已有的测试机器逻辑推理能力的数据集中缺乏这种解释信息。针对该问题，创建了一个可解释性逻辑推理的中英文数据集（explainable logical reasoning，Ex-LoR），该数据集包含3?411个逻辑推理问题与解释数据，并按照推理方法将这些问题分为六类。共设计两个任务：逻辑推理问答任务和解释生成任务。利用多个语言模型在该数据集上进行实验与分析，实验结果表明，现有语言模型尚不能很好地对逻辑推理问题进行解答并生成合理的解释，因此让机器掌握逻辑推理能力具有一定的挑战性。提出的逻辑推理数据集与实验结果可作为后续研究的基准。

关键词: 逻辑推理, 中英文数据集, 可解释性, 自然语言处理

Abstract: Logical reasoning ability is crucial to understand natural language by machines and humans. The explanation of logical reasoning problems is an elaboration and description of the logical reasoning process. However, such explanations are lacking in current logical reasoning benchmarks. For this problem, this paper creates a Chinese and English dataset called explainable logical reasoning (Ex-LoR). This dataset contains 3411 logical reasoning problems with explanation data, and categorizes these problems into 6 classes according to their reasoning methods. This paper designs two tasks：logical reasoning question and answer task, and explanation generation task. Subsequently, this paper conducts experiments and analysis on this dataset by using several language models. The results show that the existing language models are still unable to well answer logical reasoning questions and generate reasonable explanations. Therefore, it is challenging to equip machines with logical reasoning capabilities. The logical reasoning dataset and experimental results presented in this paper can be used as a benchmark for subsequent research.

Key words: logical reasoning, Chinese and English dataset, explainable, natural language processing

肖宇, 肖菁, 林桂锦, 倪荣森, 冼嘉荣, 袁基保. 可解释性逻辑推理数据集的构建和研究[J]. 计算机工程与应用, 2025, 61(4): 114-121.

XIAO Yu, XIAO Jing, LIN Guijin, NI Rongsen, XIAN Jiarong, YUAN Jibao. Construction and Study of Explainable Logical Reasoning Dataset[J]. Computer Engineering and Applications, 2025, 61(4): 114-121.

参考文献

[1] YU W, JIANG Z, DONG Y, et al. ReClor: a reading comprehension dataset requiring logical reasoning[J]. arXiv:2002.04326, 2020.
[2] LIU J, CUI L, LIU H, et al. LogiQA: a challenge dataset for machine reading comprehension with logical reasoning[C]//Proceedings of the 29th International Joint Conference on Artificial Intelligence, 2020: 3622-3628.
[3] LIU H, LIU J, CUI L, et al. LogiQA 2. 0—an improved dataset for logical reasoning in natural language understanding[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023, 31: 2947-2962.
[4] HUANG Y, FANG M, CAO Y, et al. DAGN: discourse-aware graph network for logical reasoning[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021: 5848-5855.
[5] LI X, CHENG G, CHEN Z, et al. AdaLoGN: adaptive logic graph network for reasoning-based machine reading comprehension[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022: 7147-7161.
[6] XU F, LIU J, LIN Q, et al. Logiformer: a two-branch graph transformer network for interpretable logical reasoning[C]//Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022: 1055-1065.
[7] JIAO F, GUO Y, SONG X, et al. MERIt: meta-path guided contrastive learning for logical reasoning[J]. arXiv:2203.00357, 2022.
[8] WANG S, ZHONG W, TANG D, et al. Logic-driven context extension and data augmentation for logical reasoning of text[C]//Findings of the Association for Computational Linguistics: ACL 2022, 2022: 1619-1629.
[9] 潘武, 高云鹏. 论MBA、公务员逻辑考试与批判性思维课程教学[J]. 黑河学刊, 2009（7）: 26-27.
PAN W, GAO Y P. On teaching MBA and civil service logic exams and critical thinking courses[J]. Heihe Journal, 2009(7): 26-27.
[10] WIEGREFFE S, MARASOVI A. Teach me to explain: a review of datasets for explainable natural language processing[J]. arXiv:2102.12060, 2021.
[11] KOCIJAN V, LUKASIEWICZ T, DAVIS E, et al. A review of Winograd schema challenge datasets and approaches[J]. arXiv:2004.13831, 2020.
[12] TALMOR A, HERZIG J, LOURIE N, et al. CommonsenseQA: a question answering challenge targeting commonsense knowledge[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019: 4149-4158.
[13] GORDON A, KOZAREVA Z, ROEMMELE M. SemEval-2012 task 7: choice of plausible alternatives: an evaluation of commonsense causal reasoning[C]//Proceedings of the 6th International Workshop on Semantic Evaluation, 2012: 394-398.
[14] BOWMAN S R, ANGELI G, POTTS C, et al. A large annotated corpus for learning natural language inference[J]. arXiv:1508.05326, 2015.
[15] ZHANG H, ZHAO X, SONG Y. WinoWhy: a deep diagnosis of essential commonsense knowledge for answering Winograd schema challenge[J]. arXiv:2005.05763, 2020.
[16] DU L, DING X, XIONG K, et al. e-care: a new dataset for exploring explainable causal reasoning[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022: 432-446.
[17] CHEN J J, XU R, FU Z Q, et al. E-KAR: a benchmark for rationalizing natural language analogical reasoning[C]//Findings of the Association for Computational Linguistics: ACL 2022, 2022: 3941-3955.
[18] PRASAD R, DINESH N, LEE A, et al. The Penn discourse TreeBank 2.0[C]//Proceedings of the 6th International Conference on Language Resources and Evaluation, 2008.
[19] ZELLERS R, HOLTZMAN A, BISK Y, et al. HellaSwag: can a machine really finish your sentence[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019: 4791-4800.
[20] GURURANGAN S, SWAYAMDIPTA S, LEVY O, et al. Annotation artifacts in natural language inference data[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, 2018: 107-112.
[21] POLIAK A, NARADOWSKY J, HALDAR A, et al. Hypothesis only baselines in natural language inference[C]//Proceedings of the 7th Joint Conference on Lexical and Computational Semantics, 2018: 180-191.
[22] CHIN-YEW L. Rouge: a package for automatic evaluation of summaries[C]//Proceedings of the Workshop on Text Summarization Branches Out, 2004: 74-81.
[23] PAPINENI K. BLEU: a method for automatic evaluation of MT[C]//Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, 2002: 311-318.
[24] DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, 2019: 4171-4186.
[25] YANG Z, DAI Z, YANG Y, et al. XLNet: generalized autoregressive pretraining for language understanding[C]//Advances in Neural Information Processing Systems 32, 2019.
[26] LIU Y, OTT M, GOYAL N, et al. RoBERTa: a robustly optimized bert pretraining approach[J]. arXiv:1907.11692, 2019.
[27] LEWIS M, LIU Y, GOYAL N, et al. BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 7871-7880.
[28] RAFFEL C, SHAZEER N, ROBERTS A, et al. Exploring the limits of transfer learning with a unified text-to-text transformer[J]. The Journal of Machine Learning Research, 2020, 21(1): 5485-5551.