关键词: MapReduce, 推测执行, 异构环境, K-means算法

Abstract: Aiming at the shortage of Hadoop default scheduling algorithm and LATE scheduling algorithm of heterogeneous environment, this paper proposes an enhanced adaptive MapReduce scheduling algorithm on the basis of SAMR scheduling algorithm. The algorithm records the history information of each node, and uses K-means clustering algorithm to dynamically adjust the progress value, aims to find the slow tasks which are really need begin back-up. Finally, the experimental results show that the enhanced MapReduce scheduling algorithm has some validity in the aspect of improving the estimation error of the tasks’ execution time and accurately identifying the slow tasks.

Key words: MapReduce, speculative execution, heterogeneous environment, K-means algorithm