Application for particle swarm algorithm in distributed ETL job scheduling

Abstract

Abstract: With the increasing complexity of distributed data environment, ETL tools face the challenge of many data sources, geographic distribution, massive data and other factors. The original centralized ETL workflow optimization theory can not meet the demands of the environment of the complex data. This paper presents how the discrete particle swarm optimization based on replacement is used in task scheduling of the distributed ETL. The main contents include the abstraction of the ETL task scheduling model, design of the algorithm coding, selection of objective function and so on. The realization and pseudocode of distributed ETL job scheduling strategy are also mentioned. The theory and experiment prove it to be feasible and efficient.

Key words: distributed Extration-Transformation-Loading（ETL）, task scheduling, discrete particle swarm optimization based on replacement algorithm

摘要： 随着分布式数据环境越来越复杂，ETL工具要面临数据源多、分布地域广和海量数据等因素带来的挑战。原有的集中式ETL工作流优化理论不能满足现在复杂数据环境的要求。介绍了如何将基于置换的离散型粒子群算法应用到分布式ETL任务优化调度问题上，主要工作围绕ETL工作调度模型、算法编码设计、目标函数选择等内容来展开，给出了分布式ETL工作调度策略的实现过程和伪代码。理论分析和实验证明了实际应用的有效可行性。

关键词: 分布式抽取-转换-加载（ETL）, 任务调度, 基于置换的离散型粒子群算法

WANG Chunyang, ZHAO Shuliang, WANG Changbin. Application for particle swarm algorithm in distributed ETL job scheduling[J]. Computer Engineering and Applications, 2013, 49(9): 150-155.

王春阳，赵书良，王长宾. 粒子群算法在分布式ETL任务调度中的应用[J]. 计算机工程与应用, 2013, 49(9): 150-155.

[1]	ZHANG Chi, WANG Yuxin, FENG Zhen, GUO He. Energy-Efficient Task Scheduling Algorithm in Two-Layer Virtualized Cloud Architecture [J]. Computer Engineering and Applications, 2021, 57(3): 103-111.
[2]	TIAN Zhuojing, HUANG Zhenchun, ZHANG Yinong. Review of Task Scheduling Methods in Cloud Computing Environment [J]. Computer Engineering and Applications, 2021, 57(2): 1-11.
[3]	XIANG Min, DAI Keyu, ZHOU En, LIU Yu, LEI Rujie. Scheduling Strategy for IoT Terminals Considering Task Relevance [J]. Computer Engineering and Applications, 2020, 56(23): 95-102.
[4]	YU Qiaoyu，WU Yaohua，WANG Yanyan. Tasks Scheduling Optimization of Tier-to-Tier Shuttle-Based Storage and Retrieval System with Double Lifts [J]. Computer Engineering and Applications, 2020, 56(20): 270-278.
[5]	WU Bo, XU Daoqiang, ZOU Yunfeng, WANG Tiantian, LI Xin. Stage-Aware Geo-Distributed Data Analytics Job Insurance Mechanism [J]. Computer Engineering and Applications, 2019, 55(23): 78-85.
[6]	ZHU Yongchao, ZHOU Chuan, GUO Jian, WU Yifei, CUI Yuwei. New Primary/Backup Scheduling Algorithm in Heterogeneous Distributed Systems [J]. Computer Engineering and Applications, 2019, 55(21): 93-98.
[7]	TANG Linyu, JIANG Jiafu, GU Ke. Computing Resource Allocation Scheme Based on Fog Computing [J]. Computer Engineering and Applications, 2019, 55(19): 96-104.
[8]	JIA Liyun, ZHANG Xiangli, ZHANG Hongmei. Heuristic task scheduling algorithm for distributed systems [J]. Computer Engineering and Applications, 2017, 53(12): 63-69.
[9]	WANG Xianghua1, CHEN Tefang1, ZHANG Biming2, YAN Jian1. Web data clustering algorithm with time series and task scheduling [J]. Computer Engineering and Applications, 2016, 52(9): 159-163.
[10]	LIN Ruonan, LI Qiang. Template-based task scheduling strategy and algorithm in cloud computing environment [J]. Computer Engineering and Applications, 2016, 52(19): 48-52.
[11]	WANG Bo, ZHANG Xiaolei. Task scheduling algorithm based on Particle Swarm Optimization Genetic Algorithms in cloud computing environment [J]. Computer Engineering and Applications, 2015, 51(6): 84-88.
[12]	SUN Lingyu1, LENG Ming1，2, ZHU Ping1, LI Jinzhong1. Research of dependent task schedule of cloud computing based on weighted directed hypergraph [J]. Computer Engineering and Applications, 2015, 51(24): 34-39.
[13]	WANG Zhenchao. Integration of DEA and Taguchi method of multi-task scheduling scheme in cloud computing [J]. Computer Engineering and Applications, 2015, 51(2): 140-145.
[14]	LI Jingmei, ZHANG Dahu, WU Yanxia. Heterogeneous Chip Multi-core Processor task scheduling method based on ant colony algorithm [J]. Computer Engineering and Applications, 2015, 51(18): 47-51.
[15]	LI Zhaojun1，2, LI Zheng2, WANG Xicheng2, LI Keqiu1. Dynamic cloud task scheduling method based on Kriging surrogate model [J]. Computer Engineering and Applications, 2015, 51(16): 30-35.

Application for particle swarm algorithm in distributed ETL job scheduling

粒子群算法在分布式ETL任务调度中的应用

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics