计算机工程与应用 ›› 2015, Vol. 51 ›› Issue (19): 61-65.

• 理论研究、研发设计 • 上一篇    下一篇

Hadoop平台下计算能力调度算法的改进与实现

戴小平,张宜力   

  1. 安徽工业大学 计算机学院,安徽 马鞍山 243002
  • 出版日期:2015-09-30 发布日期:2015-10-13

Realization and improvement of capacity scheduling algorithm with Hadoop platform

DAI Xiaoping, ZHANG Yili   

  1. Computer Science School, Anhui University of Technology, Ma’anshan, Anhui 243002, China
  • Online:2015-09-30 Published:2015-10-13

摘要: 在计算能力调度算法中没有全面考虑各资源特征的分配是否满足作业多样的服务要求,提出一种基于优先级的计算能力加权调度算法,根据作业的优先级以及提交时间等因素来计算作业的权重。依据作业的权重对作业队列进行排序并分配空闲的slot给队首的作业,从而避免调度陷入局部最优也能更好地满足作业的多样性服务要求。在搭建的Hadoop平台上进行实验表明,改进后的算法能较均衡地分配系统资源减少一些作业的等待时间,并且运行全部作业的用时有所减少。

关键词: 云计算, 作业调度, Hadoop, 加权, MapReduce编程模型

Abstract: Among capacity scheduling algorithms, it is not fully taken into account whether the allocation of resources meets job varied service requirements. This paper presents a priority based weighted capacity scheduling algorithm which calculates the weight based on the priority of jobs, job’s wait time and other factors of job. Based on the weight of the job to sort the job queue and assign free slot to the team’s first job, so as to avoid falling into local optimal solution and better meet job varied service requirements. Experimental results on Hadoop platform show that improved algorithm can be more balanced with the resources, reduce waiting time for some jobs, and decrease the total run time of the job.

Key words: cloud computing, job scheduling, Hadoop, weighting, MaPreduce programming model