Research on Classification of Tourist Questions Combined with Multi-head Attention Mechanism

doi:10.3778/j.issn.1002-8331.2008-0151

Abstract

Abstract: Tourism questions have the characteristics of short length and not strictly following grammatical rules, which leads to too little information capacity of the text data and serious colloquial. It is an important challenge to fully understand the semantics of question expression to improve the performance of tourist question classifiers. To this end, a tourist question classification model combining Bi-GRU, CNN, and Multi-Head-Attention is proposed. The model fuses pre-trained word vectors and semantic information processed by Bi-GRU to learn question dependency, then extracts features through CNN and Multi-Head-Attention to strengthen the learning of local features, and finally completes classification through Softmax. The F1 score of this model achieves 92.11% in the classification task of tourism questions with less text information and irregular expression, which is superior to the existing mainstream classification model.

Key words: natural language processing, tourism question classification, bidirectional gated recurrent unit（Bi-GRU）, convolutional neural network（CNN）, Multi-Head-Attention

摘要： 旅游问句具有长度较短，不严格按照语法规则的特点，导致该文本数据信息容量过少、口语化严重。充分理解问句表达的语义是提高旅游问句分类器性能面临的重要挑战，基于此，提出一个融合Bi-GRU、CNN与Multi-Head-Attention的旅游问句分类模型。该模型将预先训练的词向量和经Bi-GRU处理得到的语义信息进行融合，进行问句依赖关系学习，通过CNN和Multi-Head-Attention进行特征提取，以加强局部特征的学习，通过Softmax完成分类。实验结果表明，该模型在文本信息少、表述不规范的旅游问句分类任务中F1值达到了92.11%，优于现有的主流分类模型。

关键词: 自然语言处理, 旅游问句分类, 双向门控循环单元（Bi-GRU）, 卷积神经网络（CNN）, 多头注意力机制

WU Di, JIANG Liting, WANG Lulu, Tuergen Yibulayin, Aishan Wumaier, Zaokere Kadder. Research on Classification of Tourist Questions Combined with Multi-head Attention Mechanism[J]. Computer Engineering and Applications, 2022, 58(3): 165-171.

吴迪, 姜丽婷, 王路路, 吐尔根·依布拉音, 艾山·吾买尔, 早克热·卡德尔. 结合多头注意力机制的旅游问句分类研究[J]. 计算机工程与应用, 2022, 58(3): 165-171.

References

[1] PROVENZANO D，BAGGIO R.A complex network analysis of inbound tourism in sicily[J].International Journal of Tourism Research，2020，22（4）.
[2] 郑实福，刘挺，秦兵，等.自动问答综述[J].中文信息学报，2002，16（6）：46-52.
ZHENG S F，LIU T，QIN B，et al.Overview of question-answering[J].Journal of Chinese Information Processing，2002，16（6）：46-52.
[3] CHUNG J，GULCEHRE C，CHO K H，et al.Empirical evaluation of gated recurrent neural networks on sequence modeling[J].arXiv：1412.3555，2014.
[4] KIM Y.Convolutional neural networks for sentence classification[J].arXiv：1408.5882，2014.
[5] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Neural Information Processing Systems，2017：5998-6008.
[6] ZHANG W，DUAN L，CHEN J.Study on Chinese question classification based on SVM multi-category classification[C]//Fourth International Conference on Machine Vision（ICMV 2011）：Machine Vision，Image Processing，and Pattern Analysis，2012.
[7] JIANG L，LI C，WANG S，et al.Deep feature weighting for naive Bayes and its application to text classification[J].Engineering Applications of Artificial Intelligence，2016，52：26-39.
[8] JIA B B，ZHANG M L.Multi-dimensional classification via kNN feature augmentation[J].Pattern Recognition，2020，106：107423.
[9] BAE K，KO Y.Efficient question classification and retrieval using category information and word embedding on cQA services[J].Journal of Intelligent Information Systems，2019，53（1）：27-49.
[10] 杨志明，王来奇，王泳.基于双通道卷积神经网络的问句意图分类研究[J].中文信息学报，2019，33（5）：122-131.
YANG Z M，WANG L Q，WANG Y.Questions intent classification based on dual channel convolutional neural network[J].Journal of Chinese Information Processing，2019，33（5）：122-131.
[11] XIA W，ZHU W，LIAO B，et al.Novel architecture for long short-term memory used in question classification[J].Neurocomputing，2018，299：20-31.
[12] 姚苗，杨文忠，袁婷婷，等.自注意力机制的短文本分类算法[J].计算机工程与设计，2020（6）：1592-1598.
YAO M，YANG W Z，YUAN T T，et al.Short text classification algorithm of self-attention mechanism[J].Computer Engineering and Design，2020（6）：1592-1598.
[13] LIU J，YANG Y，LV S，et al.Attention-based BiGRU-CNN for Chinese question classification[J].Journal of Ambient Intelligence and Humanized Computing，2019：1-12.
[14] BANERJEE S，NASKAR S K，ROSSO P，et al.Classifier combination approach for question classification for Bengali question answering system[J].Sadhana-Academy Proceedings in Engineering Sciences，2019，44（12）：1-14.
[15] 梁志剑，谢红宇，安卫钢.基于BiGRU和贝叶斯分类器的文本分类[J].计算机工程与设计，2020，41（2）：381-385.
LIANG Z J，XIE H Y，AN W G.Text classification based on bidirectional GRU and Bayesian classifier[J].Computer Engineering and Design，2020，41（2）：381-385.
[16] MOHAMMED M，OMAR N.Question classification based on Bloom’s taxonomy cognitive domain using modified TF-IDF and word2vec[J].Plos One，2020，15（3）.
[17] SOMNATH B，SUDIP N，PAOLO R，et al.Code mixed cross script factoid question classification-a deep learning approach[J].Journal of Intelligent and Fuzzy Systems，2018，34（5）：2959-2969.
[18] MIKOLOV T，CHEN K，CORRADO G S，et al.Efficient estimation of word representations in vector space[C]//International Conference on Learning Representations，2013.
[19] ZAREMBA W，SUTSKEVER I，VINYALS O.Recurrent neural network regularization[J].arXiv：1409.2329，2014.
[20] STAUDEMEYER R C，MORRIS E R.Understanding LSTM-a tutorial into long short-term memory recurrent neural networks[J].arXiv：1909.09586，2019.
[21] MNIH V，HEESS N，GRAVES A，et al.Recurrent models of visual attention[J].arXiv：1406.6247，2014.