Shuffle optimization for Spark cluster

doi:10.3778/j.issn.1002-8331.1701-0238

Computer Engineering and Applications ›› 2018, Vol. 54 ›› Issue (4): 72-76.DOI: 10.3778/j.issn.1002-8331.1701-0238

Previous Articles Next Articles

Shuffle optimization for Spark cluster

XIONG Anping1，2, XIA Yuchong1, YANG Fangfang1

1.School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
2.Chongqing Engineering Research Center of Mobile Internet Data Application, Chongqing 400065, China

Online:2018-02-15 Published:2018-03-07

一种Spark集群下的shuffle优化机制

熊安萍1，2，夏玉冲1，杨方方1

1.重庆邮电大学计算机科学与技术学院，重庆 400065
2.重庆市移动互联网数据应用工程技术研究中心，重庆 400065

Abstract

Abstract: Spark is a distributed processing framework based on memory. The large amounts of data generated by the shuffle process deeply affect the network transmission, which has become one of the main bottlenecks of the Spark performance. In order to solve the problem of unbalanced data distribution resulting in the I/O load imbalance in different nodes, a restart policy based on task local level is designed. Finally, the optimization mechanism is verified by experiments, which can reduce the execution time of task and improve the efficiency of shuffle process.

Key words: Spark cluster, shuffle process, data transfer, locality, schedule strategy

摘要： Spark是基于内存的分布式数据处理框架，其shuffle过程中大量数据需要通过网络传输，已成为Spark最主要的瓶颈之一。针对shuffle过程中存在的数据分布不均造成不同节点网络I/O负载不均的问题，设计了基于task本地性等级的重启策略，进一步提出了均衡的调度策略来平衡各节点的网络I/O负载。最后通过实验验证了优化机制能够减少计算任务的执行时间，提升整个shuffle过程的执行效率。

关键词: Spark集群, shuffle过程, 数据传输, 本地性, 调度策略

XIONG Anping1，2, XIA Yuchong1, YANG Fangfang1. Shuffle optimization for Spark cluster[J]. Computer Engineering and Applications, 2018, 54(4): 72-76.

熊安萍1，2，夏玉冲1，杨方方1. 一种Spark集群下的shuffle优化机制[J]. 计算机工程与应用, 2018, 54(4): 72-76.

[1]	DING Zhihui, QIAO Gangzhu, CHENG Tan, SU Rong. Shapelets Transform Method Based on LSH [J]. Computer Engineering and Applications, 2021, 57(3): 112-119.
[2]	XIONG Lin, TANG Wanmei. Incremental Learning Algorithm Based on Heterogeneous Classifier Ensemble [J]. Computer Engineering and Applications, 2020, 56(7): 155-161.
[3]	JIANG Yirui, PEI Yang, CHEN Lei, WANG Wenle, DAI Jiangyan, YI Yugen. Multiple Locality-Constrained Self-Representation for Spectral Clustering [J]. Computer Engineering and Applications, 2020, 56(11): 172-178.
[4]	MA Lihong, TAN Xueshi. Zero-Shot Classification with Manifold Regularization Graph Based on Common Structure Assumption [J]. Computer Engineering and Applications, 2019, 55(15): 153-160.
[5]	CHEN Sisi1，2, ZENG Shan1, LI Weidong1，2, QI Fazhi1. Studies and Implementation of Data Transfer System for High Energy Physics Experiment [J]. Computer Engineering and Applications, 2019, 55(13): 218-224.
[6]	XIANG Chengyu, WANG Dongli, ZHOU Yan, LI Yafang. Image classification based on RGB-D fusion feature [J]. Computer Engineering and Applications, 2018, 54(8): 178-182.
[7]	WANG Cong1，2, XU Qi1，2, CHENG Yaodong1, CHEN Gang1. Research of event data management and data transfer system in high energy physics [J]. Computer Engineering and Applications, 2018, 54(23): 230-237.
[8]	CHEN Yuqi1, LEI Gang1, YAO Minghai2, YI Yugen1. Locality constrained adaptive graph based label propagation approach [J]. Computer Engineering and Applications, 2018, 54(20): 14-19.
[9]	XU Zhengguo, YAO Jiaqi, ZHENG Hui. Research on anti-noise features of network protocol classification [J]. Computer Engineering and Applications, 2018, 54(17): 95-102.
[10]	TANG Chunming, ZHAO Hongbo, ZHANG Xiaoyu. East Asian facial expression recognition algorithm based on manifold learning 2D-LDLPA [J]. Computer Engineering and Applications, 2018, 54(17): 146-150.
[11]	CAO Jie1，2, ZHU Jingjing1, LI Wei3, WANG Jinhua3. Face recognition based on fusing LSH and LoG feature [J]. Computer Engineering and Applications, 2018, 54(17): 158-163.
[12]	WEI Baoguo, ZHAO Sitong, WEN Xuliang. Multi-region tracking based on local histogram [J]. Computer Engineering and Applications, 2018, 54(14): 26-33.
[13]	CHEN Ruimin1，2, SUN Shengli1, LIAO Xingxing1. Novel method for dynamic hand gesture recognition based on Temporal Locality Sensitive Histograms of Oriented Gradients [J]. Computer Engineering and Applications, 2018, 54(13): 142-147.
[14]	LUO Jun, LI Jinhua. LSHBMRPK-means algorithm and its application. Computer Engineering and Applications [J]. Computer Engineering and Applications, 2017, 53(21): 62-67.
[15]	ZHU Xiangbin, QIU Huiling. Human activity recognition with smartphone sensor data [J]. Computer Engineering and Applications, 2016, 52(23): 1-5.

Shuffle optimization for Spark cluster

一种Spark集群下的shuffle优化机制

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics