关键词: 列变换, 并行互信息计算, 分类数据, Spark平台

Abstract:

To resolve the problem of large amount of mutual information calculation for large-scale categorical data, this paper proposes a Parallel Mutual information calculation method for categorical data（PMS）, which is based on the Spark memory computing platform. This algorithm first uses column transformation to transform the data set into multiple data subsets. And then, PMS uses two variable-length arrays to cache intermediate results, solving the problem of large amount of calculation and strong repeatability in categorical data mutual information calculation. Finally, PMS algorithm is implemented and evaluated in a Spark cluster equipped with 24 computing nodes using artificial and real data sets. Experimental results verify that PMS algorithm achieves high performance in terms of efficiency, scalability and scalability.

Key words: column-wise transformation, Parallel Mutual-information computation, categorical data, Spark platform