关键词: Transformer, 掩码语言模型, 对比学习, 微调, 虚假关联, 泛化能力

Abstract: Autoattention networks play an important role in a language model based on Transformer, where a fully connected structure can capture non-continuous dependencies in a sequence in parallel. However, the fully connected self-attention network is easy to overfit to false association information, such as false association between words and words, and between words and the predicted target. This overfitting problem limits the ability of language models to generalize data outside the domain or the distribution. To improve the robustness and generalization ability of the Transformer language model against false associations, fine-tuning framework via mask language model enhanced representations based contrastive learning is proposed in this paper. Specifically, the text sequence and the sequence after its random mask are sent into a twin network, and then the model parameters are learned by combining the contrast learning objective and the downstream task objective. Each twin network consists of a pre-trained language model and a task classifier. Therefore, the fine-tuning framework is more consistent with the mask language model pre-training learning mode and can maintain the generalization ability of pre-training knowledge in downstream tasks. The MNLI, FEVER, and QQP datasets and their challenge datasets are compared with the latest baseline models, including large language models ChatGPT, GPT4, and LLaMA. Experimental results show that the proposed model can guarantee the performance in distribution and improve the performance out of distribution. The experimental results on ATIS and Snips data sets prove that the model is also effective in common natural language processing tasks.

Key words: Transformer, masked language model, contrast learning, fine-tuning, spurious association, generalization ability