关键词: 大数据, k-means, 局部敏感哈希函数, MapReduce, 推荐算法

Abstract: To deal with the problems that time complexity is extremely high and the result of clustering ispoor when basic k-means algorithm is used to handle on big data issues, the paper proposes LSHBMRPK-means algorithm, locality sensitive hashing-based MapReduce parallelized k-means algorithm. Due to the scalability problem of recommendation system, the paper applies LSHBMRPK-means algorithm to cluster-based collaborative filtering algorithm. In addition, to handle on the issue of sparsity in the rating dataset, the paper uses the method of LFM to fill in the sparse rating dataset, and proposes LFM-based LSHBMRPK-means collaborative filtering algorithm. Primary experiments show that LSHBMRPK-means can improve the efficiency and quality of clustering, the proposed algorithm combined with the filtering algorithm has a good scalability, and at the same time it has solved the problem of poor clustering quality caused by the sparse rating dataset.

Key words: big data, k-means, locality sensitive hashing function, MapReduce, recommendation algorithm