计算机工程与应用 ›› 2022, Vol. 58 ›› Issue (18): 327-334.DOI: 10.3778/j.issn.1002-8331.2102-0275

• 工程与应用 • 上一篇    

异构集群节点与作业特性感知资源分配算法

胡亚红,吴寅超,朱正东,李小轩   

  1. 1.浙江工业大学 计算机学院,杭州 310023
    2.西安交通大学 计算机学院,西安 710049
  • 出版日期:2022-09-15 发布日期:2022-09-15

Heterogeneous Cluster Resource Allocation Algorithm Considering Application  and Node Characteristics

HU Yahong, WU Yinchao, ZHU Zhengdong, LI Xiaoxuan   

  1. 1.College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou 310023, China
    2.School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an 710049, China
  • Online:2022-09-15 Published:2022-09-15

摘要: 为了避免多应用间的资源争用,Spark采用了FIFO、FAIR等作业调度策略,辅以SpreadOut和非SpreadOut两种资源调度算法,但是这些算法没有充分考虑用户作业类型和集群节点性能的相互关系。用户作业类型及节点性能偏向感知的资源调度算法ATNPA提出了对该问题的解决方案。ATNPA根据作业运行所需的内存量和CPU核数将用户作业分为CPU密集型和内存密集型。节点的性能偏向性由节点的静态因素和动态因素决定。静态因素包括CPU速度、内存大小、CPU核数和磁盘容量等;动态因素包括CPU剩余率、内存剩余率、磁盘剩余率和磁盘读写速度等。ATNPA算法在进行资源分配时,能够将作业分配到最适合其类型的节点上。仿真实验表明,与未考虑节点和作业匹配的算法相比较,ATNPA能够有效缩短作业的执行时间、提高集群的性能。

关键词: 资源调度, 任务调度, 作业类型, 节点性能偏向, Spark

Abstract: To avoid resource contention among multiple applications, FIFO and FAIR are adopted by Spark, together with two resource scheduling algorithms, i.e., SpreadOut and non-SpreadOut. However, the existing algorithms do not fully consider the relationship between the user application type and the node features. To handle this issue, ATNPA(application type and node preference-aware scheduling algorithm), an algorithm that comprehensively considers this relationship is proposed. According to its memory and CPU requirements, an application is classified into two categories, i.e., CPU intensive and memory intensive. The node preference is decided by both static attributes and dynamic attributes. The static attributes refer to the inherent features of a node, such as the memory size, the number of CPU cores, etc. The dynamic attributes are the features that may change with the working status of the node, such as the residual memory and CPU, etc. Therefore, ATNPA can allocate an application to the nodes that are most suitable to its type during the resource allocation period. Experimental results validate that compared with scheduling algorithms without considering the node preference, ATNPA can reduce the application execution time and improve system performance.

Key words: resource scheduling, application scheduling, application type, node preference type, Spark