%0 Journal Article %A LI Junli %T Parallel Mutual-Information Computation of Categorical Data Based on Spark %D 2021 %R 10.3778/j.issn.1002-8331.2003-0432 %J Computer Engineering and Applications %P 95-100 %V 57 %N 7 %X

To resolve the problem of large amount of mutual information calculation for large-scale categorical data, this paper proposes a Parallel Mutual information calculation method for categorical data(PMS), which is based on the Spark memory computing platform. This algorithm first uses column transformation to transform the data set into multiple data subsets. And then, PMS uses two variable-length arrays to cache intermediate results, solving the problem of large amount of calculation and strong repeatability in categorical data mutual information calculation. Finally, PMS algorithm is implemented and evaluated in a Spark cluster equipped with 24 computing nodes using artificial and real data sets. Experimental results verify that PMS algorithm achieves high performance in terms of efficiency, scalability and scalability.

%U http://cea.ceaj.org/EN/10.3778/j.issn.1002-8331.2003-0432