Method for computing execution time of jobs in homogeneous hadoop environments

Computer Engineering and Applications ›› 2014, Vol. 50 ›› Issue (10): 249-252.

Previous Articles Next Articles

Method for computing execution time of jobs in homogeneous hadoop environments

ZHANG Xiaohong1，2, HAI Linpeng2, JIA Zongpu2, SHEN Jiquan3, ZHAO Wentao2

1.Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, Guangdong 518055, China
2.School of Computer Science and Technology, Henan Polytechnic University, Jiaozuo, Henan 454003, China
3.Center of Modern Education, Henan Polytechnic University, Jiaozuo, Henan 454003, China

Online:2014-05-15 Published:2014-05-14

同构Hadoop环境作业执行时间计算方法

张霄宏1，2，海林鹏2，贾宗璞2，沈记全3，赵文涛2

1.中国科学院深圳先进技术研究院，广东深圳 518055
2.河南理工大学计算机科学与技术学院，河南焦作 454003
3.河南理工大学现代教育中心，河南焦作 454003

Abstract

Abstract: Execution time is very important for job scheduling. In this paper, the execution characters of Hadoop MapReduce jobs are analyzed, and then a new method is proposed to compute the execution times of these jobs. The method takes the execution times of map task and reduce task as input data. It captures these execution times by pre-executing under an assumption. The method has been evaluated in a Linux cluster, the experiment results show that the method computed the execution times of jobs with the error rate no more than 7%.

Key words: Hadoop MapReduce, execution time, scheduling

摘要： 执行时间是作业调度的重要参考因素之一。通过分析Hadoop MapReduce环境作业的执行特征，提出了以map任务和reduce任务执行时间为输入，估算作业执行时间的方法。该方法在一定假设条件下，借助作业预执行来获取map任务和reduce任务的执行时间。实验结果表明，该方法估算作业执行时间的误差率小于7%。

关键词: Hadoop MapReduce, 作业执行时间, 调度

ZHANG Xiaohong1，2, HAI Linpeng2, JIA Zongpu2, SHEN Jiquan3, ZHAO Wentao2. Method for computing execution time of jobs in homogeneous hadoop environments[J]. Computer Engineering and Applications, 2014, 50(10): 249-252.

张霄宏1，2，海林鹏2，贾宗璞2，沈记全3，赵文涛2. 同构Hadoop环境作业执行时间计算方法[J]. 计算机工程与应用, 2014, 50(10): 249-252.

[1]	XIE Ruiqiang, ZHANG Huizhen. Two-Vector Wolf Pack Algorithm for Flexible Job Shop Scheduling Problem [J]. Computer Engineering and Applications, 2021, 57(7): 251-256.
[2]	WANG Yuncheng, ZHOU Chunhua, CHEN Chuxiang, WU Shanming, CHEN Bing. Extensible Intelligent Scheduling System Based on Two Fork Knowledge Tree Reasoning [J]. Computer Engineering and Applications, 2021, 57(5): 251-257.
[3]	ZHANG Chi, WANG Yuxin, FENG Zhen, GUO He. Energy-Efficient Task Scheduling Algorithm in Two-Layer Virtualized Cloud Architecture [J]. Computer Engineering and Applications, 2021, 57(3): 103-111.
[4]	LI Baoshuai, YE Chunming. Job Shop Scheduling Problem Based on Deep Reinforcement Learning [J]. Computer Engineering and Applications, 2021, 57(23): 248-254.
[5]	CAO Lijia, LIU Yang. Recent Advances in Scheduling Optimization of Automated Guided Vehicles in Manufacturing Workshops [J]. Computer Engineering and Applications, 2021, 57(21): 59-67.
[6]	TIAN Zhuojing, HUANG Zhenchun, ZHANG Yinong. Review of Task Scheduling Methods in Cloud Computing Environment [J]. Computer Engineering and Applications, 2021, 57(2): 1-11.
[7]	ZHOU Yamin, YANG Yongsheng, ZHONG Meisu, WANG Yusheng, XU Xueqian. Scheduling of Crossing Double ASC in Yard of Automated Terminals [J]. Computer Engineering and Applications, 2021, 57(2): 251-256.
[8]	SHI Yuqiang, TIAN Yongzheng, ZHANG Yuqi, SHI Xiaoqiu. Solving FJSP by Multi-population Genetic Algorithms with Complex Network Structure [J]. Computer Engineering and Applications, 2021, 57(2): 257-266.
[9]	ZHOU Yuanling, HU Xiaobing, JIANG Daiyu, LI Hang. Research on Optimization Algorithm of Workshop Scheduling Based on Improved NSGA-II [J]. Computer Engineering and Applications, 2021, 57(19): 274-281.
[10]	DAI Jiangtao, HAN Xiaolong. Coordinated Scheduling of Equipment in Container Terminals Considering Energy Consumption Under Different Job Status [J]. Computer Engineering and Applications, 2021, 57(19): 290-298.
[11]	GUO Jiali, WANG Qiuping, WANG Xiaofeng. Hybrid Moth Flame Algorithm with Learning Strategy and Neighboring Search [J]. Computer Engineering and Applications, 2021, 57(12): 170-179.
[12]	LI Yuxin, LI Yueyue. Multi-objective Vehicle Scheduling Problem for JIT Procurement [J]. Computer Engineering and Applications, 2021, 57(12): 263-272.
[13]	LIANG Chengji, HUANG Shuai. Scheduling Optimization of Multiple Container Blocks with Multiple Yard Cranes at Container Terminal Based on Time Window [J]. Computer Engineering and Applications, 2020, 56(8): 226-232.
[14]	YANG Yajie, CHANG Daofang, YU Fang. Multi-Resource Coordinated Scheduling of Automated Terminals Considering AGV Collision Avoidance [J]. Computer Engineering and Applications, 2020, 56(6): 246-253.
[15]	QIN Qin, LIANG Chengji. Optimization of Equipment Coordination Scheduling with Considering Buffer Space in Automated Container Terminal [J]. Computer Engineering and Applications, 2020, 56(6): 262-270.

Method for computing execution time of jobs in homogeneous hadoop environments

同构Hadoop环境作业执行时间计算方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics