关键词: 深度学习, 自然语言处理, 预处理, 词向量, 语言模型

Abstract:

In recent years, deep learning technology has been advancing, pre-training technology for deep learning brings natural language processing into a new era. Pre-training model aims to how to make pre-trained model stay in good initial state and achieve better performances in subsequent downstream tasks. This paper firstly introduces pre-training technology and its development history. And then, this paper further classifies it into the following two types, namely probability-statistics-based traditional model and deep-learning-based new model, according to different features of pre-training models to conduct corresponding detailed introductions. Firstly, it briefly analyzes the characteristics and limitations of today’s pre-training models and highlights the existing deep-learning-based pre-training models. And based on their performances in downstream tasks, it gives necessary comparisons and assessments accordingly. Finally, it combs out a series of whole-new pre-training models with instructive significances,briefly describes corresponding feasible improvement mechanisms and the performance enhancements achieved in downstream tasks, summarizes the problems existing therein, as well as prospectes its development trend in near future.

Key words: deep learning, natural language processing, pre-training, word embedding, language model