Improved LATE scheduling algorithm on Hadoop paltform

Abstract

Abstract: The existing schedulers of MapReduce on the Hadoop platform are analyzed. To consider the lack of LATE scheduling algorithm in allocating TaskTracker to execute backup tasks for backward tasks, combining with the particularity of Hadoop cluster heterogeneity and workload, an improved scheduling algorithm is proposed based on LATE scheduling algorithm. The experiments and performance analysis for the algorithm show that it has significantly improved in the completion time and load balancing.

Key words: Hadoop, MapReduce, LATE, backward task

摘要： 对Hadoop平台下的MapReduce现有的调度器进行分析研究。针对LATE调度算法在分配节点执行落后任务的备份任务时的不足，结合Hadoop集群的异构性和工作负载的特殊性，在LATE调度算法的基础上提出了一种改进的LATE调度算法。对该算法进行实验和性能分析，表明该算法在完成时间和负载均衡方面有很大改进。

关键词: Hadoop, MapReduce, LATE, 落后任务

HU Dan, YU Jiong, YING Changtian, ZOU Weiming. Improved LATE scheduling algorithm on Hadoop paltform[J]. Computer Engineering and Applications, 2014, 50(4): 86-89.

胡丹，于炯，英昌甜，邹伟明. Hadoop平台下改进的LATE调度算法[J]. 计算机工程与应用, 2014, 50(4): 86-89.

[1]	WU Wenjie, SONG Wen’ai, GAO Xuemei, YANG Jijiang, WANG Qing, HUANG Liping, LEI Yi. Review of X-Ray-Based Computer-Aided Diagnosis of Adult OSA [J]. Computer Engineering and Applications, 2021, 57(9): 1-8.
[2]	MA Qiaomei, WANG Mingjun, LIANG Haoran. License Plate Location Detection Algorithm Based on Improved YOLOv3 in Complex Scenes [J]. Computer Engineering and Applications, 2021, 57(7): 198-208.
[3]	WAN Qin, ZHU Xiaolin, CHEN Guoquan, XIAO Yueping. Research on Depth Map Restoration Algorithm Based on Hierarchical Joint Bilateral Filter [J]. Computer Engineering and Applications, 2021, 57(6): 184-190.
[4]	HAO Yaling, ZHU Xinjuan, WU Xiaojun. Research on Software Customizing System for 3D Virtual Display [J]. Computer Engineering and Applications, 2021, 57(5): 271-278.
[5]	LIAO Wenxiong, ZENG Bi, XU Yayun. Natural Language Processing Model Based on One-Dimensional Dilated Convolution and Attention Mechanism [J]. Computer Engineering and Applications, 2021, 57(4): 114-119.
[6]	LI Zixing, LAN Zhen, TANG Dengqing, YAN Chao, XIANG Xiaojia, ZHOU Han. Survey on ERP-Based Target Detection [J]. Computer Engineering and Applications, 2021, 57(23): 37-49.
[7]	ZHANG Shihao, YANG Xiujun, WU Linhuang, CHEN Pingping. Lightweight Multi-scale Attention Fusion Algorithm for License Plate Detection [J]. Computer Engineering and Applications, 2021, 57(22): 208-214.
[8]	MA Yanfang, LI Baoyu, YANG Yifu, FENG Cuiying. Two-Echelon Capacitated Vehicle Routing Model and Algorithm for Fresh Products Distribution with Customer Classification [J]. Computer Engineering and Applications, 2021, 57(20): 287-298.
[9]	DONG Lihong, LI Yuxing, FU Limei. Image Semantic Segmentation by Fusion of Global and Low Order Features [J]. Computer Engineering and Applications, 2021, 57(17): 210-216.
[10]	MENG Na, LI Yanyan, WANG Lin. Parameter Estimation for Cubic Frequency Modulated Signals Based on CITAF-HAF [J]. Computer Engineering and Applications, 2021, 57(17): 75-79.
[11]	WU Dongyang, DOU Jianping, LI Jun. Design of Digital Twin System for Quadrotor [J]. Computer Engineering and Applications, 2021, 57(16): 237-244.
[12]	TONG Wenlin, CHEN Dewang, HUANG Yunhu, LYU Yisheng. Fuzzy System Optimization Method Based on Simulated Annealing and Rule Reduction [J]. Computer Engineering and Applications, 2021, 57(16): 142-150.
[13]	LI Leixiao, DENG Dan, LI Jie, WANG Yongsheng. All-to-All Comparison Computing Data Distribution Strategy Based on Particle Swarm Optimization [J]. Computer Engineering and Applications, 2021, 57(15): 109-117.
[14]	ZHANG Kai, JIN Peng, CUI Yong. Multi-vehicle Split Pickup and Delivery Problem with Time Windows [J]. Computer Engineering and Applications, 2021, 57(14): 281-288.
[15]	ZHANG Geyi, CHEN Xiaogang, GUO Jipeng, SONG Zhitang, CHEN Bangming. Solution to Optimize PCM Write Depending on Asymmetries [J]. Computer Engineering and Applications, 2021, 57(14): 75-82.

Improved LATE scheduling algorithm on Hadoop paltform

Hadoop平台下改进的LATE调度算法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics