关键词: 聚类, LD算法, 文本相似度矩阵, 向量空间模型, 文本相似性

Abstract: In order to solve the problem of the imprecise calculation result of text similarity which comes from text similarity algorithm based on traditional vector space model, it doesn’t consider vector dimension and micro variation of key word, proposes TSABCLDA（Text Similarity Algorithm Based on Clustering and LD Algorithm） with the situation of micro variation of key word. In the present work, it makes some pretreatment of removing the number, punctuation and stop word. It reduces the low-frequency words in the text with clustering method, calculates the similarity between characteristic words by LD algorithm, builds text similarity matrix. It calculates the similarity between texts by characteristic words similarity matrix and space vector which is built by weight. It not only considers the micro variation situation of key word, but also solves the high dimensional problems of text effectively. If applied to text mining, it will improve the efficiency of mining of similarity text. The experimental results show that precise of the algorithm is improved obviously with the discovery of similarity text in situation of micro variation and a certain range of threshold values.

Key words: clustering, LD algorithm, text similarity matrix, vector space model, text similarity