Abstract:

To resolve the problem of large amount of mutual information calculation for large-scale categorical data, this paper proposes a Parallel Mutual information calculation method for categorical data（PMS）, which is based on the Spark memory computing platform. This algorithm first uses column transformation to transform the data set into multiple data subsets. And then, PMS uses two variable-length arrays to cache intermediate results, solving the problem of large amount of calculation and strong repeatability in categorical data mutual information calculation. Finally, PMS algorithm is implemented and evaluated in a Spark cluster equipped with 24 computing nodes using artificial and real data sets. Experimental results verify that PMS algorithm achieves high performance in terms of efficiency, scalability and scalability.

Key words: column-wise transformation, Parallel Mutual-information computation, categorical data, Spark platform

摘要：

针对大规模类别数据的互信息计算量非常大的问题，利用Spark内存计算平台，提出了类别数据的并行互信息计算方法，该算法首先采用列变换将数据集转换成多个数据子集；然后采用两个变长数组缓存中间结果，解决了类别数据特征对间互信息计算量大、重复性强的问题；最后在配备了24个计算节点的Spark集群中，使用人工合成和真实数据集验证了算法。实验结果表明，该算法在效率、可伸缩性和可扩展性等方面都达到了较高的性能。

关键词: 列变换, 并行互信息计算, 分类数据, Spark平台