关键词: Spark, MPI, 科学计算, 内存计算, 迭代算法

Abstract: Engineering problems and scientific research are facing dual challenges of big data processing and high-performance computing tasks. Spark, a distributed processing framework based on in-memory computing technology, has been widely used in academia and industry. However, its MapReduce-like programming model fails to communicate between tasks, causing numerical algorithms in scientific computing cannot be efficiently implemented. In response to the above problems, a computing system is proposed in this paper that combines Spark in-memory computing model with MPI message passing, which takes full advantage of the fast speed of memory access and multiple high performance communication mechanisms of MPI. It can not only supplement the insufficient expressiveness of the Spark programming model, but also provide a data-oriented DAG computation method for MPI. Internal runtime environment and scheduling strategy of Spark are modified to seamlessly integrate MPI into Spark to provide a unified in-memory computing system for high-performance computing and big data processing tasks. The tests indicate that the performance of numerical computation and iterative algorithm is improved by at least 50% compared with Spark.

Key words: Spark, MPI, scientific computing, in-memory computing, iterative algorithm