关键词: bert, transformer, encoder, CNN, 文本分类, fine-tuning, self-attention, 过拟合

Abstract:

Due to Bert-base, the parameters of Chinese pre-training model are huge, the internal parameters change little during the fine-tuning of classification task, which is prone to overfitting phenomenon and weak generalization ability. Moreover, this model is pre-training in the unit of words and contains less information of words. To solve these problems, this study proposes the BERT-TECNN model, model uses BERT-base, the Chinese model as a dynamic character vector model to output characteristic information containing the depth of character vector. The transformer encoder layer again long since the attention for the data is calculated, to extract the feature information, in order to improve the generalization ability of the model, CNN layer with different size of convolution kernels, capture different length in each data word information, finally softmax is used for classification. Compared with Word2Vec+CNN, Word2Vec+BiLSTM, Elmo+CNN, BERT+CNN, BERT+BiLSTM, BERT+Transformer and other deep learning text classification models on three data sets, and the accuracy, precision, recall rate and F1 measure values are all the highest. Experiments show that the model can effectively extract the feature information of words in text, optimize the problem of overfitting, and improve the generalization ability.

Key words: bert, transformer, encoder, CNN, text classification, fine-tuning, self-attention, overfit