关键词: 短文本数据流分类, 分布式处理, Spark环境, 概念漂移

Abstract:

Short text data streams emerging on social network platforms such as Weibo and Facebook have the characteristics of magnanimity, high-dimension, sparsity and fast variable, and it is hence a huge challenge for the short text data stream classification. Existing short text data stream classification methods are difficult to effectively solve the high-dimensional and sparse feature problem, and spend higher time costs in the processing of massive data streams. Motivated by this, a distributed fast short text data stream classification method based on Spark is proposed in this paper. On the one hand, the external corpus is used to construct the Word2vec model to solve the high-dimension and sparsity issue of short texts, and the extended word vector library is constructed to adapt to the fast variability of the texts. Then, an LR classifier integration model is proposed for classifying short text data streams. The classifier utilizes an FTRL method to implement online update of model parameters, and introduces a time factor weighting mechanism to adapt to the concept drift environment. On the other hand, the proposed method uses distributed processing to improve the performance of handling with massive short text streams. Finally, experiments conducted on three real short text data streams show that the proposed method greatly reduces the time consumption while improving the classification accuracy.

Key words: short text data stream classification, distributed processing, Spark, concept drift