关键词: 短文本, 权重计算, TF-IWF方法, 主题模型

Abstract: Short texts are normally featured with less content, looser text format, varied sentence length and relatively complex sentence structure. Consequently, the effects of traditional classification algorithms are quite unsatisfactory. This paper presents an authentic comprehensive method by the fusion of BTM theme features and well-improved weight calculation method for short text classification. In order to achieve this, two steps are in necessity. Firstly, the paper reduces the term frequency weight according to the TF-IWF. In the meantime, it introduces the word distribution probability value so that a new algorithm for computing weights will derive. Secondly, it uses the topic words of BTM topic model to complement empty documents. Meanwhile, the probability distribution of each document in each topic will be carefully selected as the document’s other features. Experimental results indicate that with the help of this newly created method, the results of F1 has been improved by around 10% compared to the original TF-IWF method.

Key words: short text, weight calculation, Inverse Word Frequency（TF-IWF）, topic model