Abstract: Fault tolerance is an inevitable issue for long-running large-scale applications. Heterogeneous devices make the reliability problem more extrude. Focusing on the hardware failures, this paper presents a fault-tolerant mechanism in the heterogeneous clusters. It’s also implemented in the large-scale parallelization software of the cosmological fluid simulation in WIGEON based on the parallel pattern of Charm++ RTS and CUDA. What’s more, it’s combined with dynamic load-balancing for adapting the unbalanced computing node configuration after failures. Through the experiments and analysis on Mole8.5, it validates the efficiency and feasibility of this algorithm, and the recovery time only takes 1~4 seconds, and it also uses the distributed redundant data to improve Double-in-Memory checkpoint algorithm. The new fault-tolerant algorithm reduces memory footprint by 50% at the most without extra performance loss.

Key words: fault tolerance, heterogeneous, in-memory checkpoint, Charm++, load-balancing, distributed redundant

摘要： 容错问题是大规模并行程序长时间运行中不可回避的问题，超级计算机中异构计算部件的加入使得该问题更加复杂。考察由CPU和GPU组成的异构并行系统中应用程序的容错，利用Charm++并行编程模型和CUDA的并行计算架构，对大规模计算宇宙学软件WIGEON进行重构。针对异构并行系统中存在的fail-stop硬件故障，设计并实现了内存检查点的应用容错机制。支持计算恢复后对产生变化的CPU/GPU资源配置进行自适应负载调整。通过在高性能计算机Mole8.5上的实验和分析，验证了异构容错方案的高效性和可行性，故障恢复时间仅需1~4 s。此外，使用分布式冗余数据改进了Charm++现有内存检查点存储模式，对比原有Double-in-Memory机制，性能未受影响，且最多降低了50%的额外内存使用量。

关键词: 容错, 异构, 无盘检查点, Charm++, 负载均衡, 分布式冗余