Research on optimized MapReduce model of Hadoop cloud platform

Computer Engineering and Applications ›› 2016, Vol. 52 ›› Issue (22): 22-25.

Previous Articles Next Articles

Research on optimized MapReduce model of Hadoop cloud platform

ZHANG Hong1，2, WANG Xiaoming1, CAO Jie2, MA Yanhong3, GUO Yirong1, WANG Min1

1.College of Electrical & Information Engineering, Lanzhou University of Technology, Lanzhou 730050, China
2.College of Computer & Communication, Lanzhou University of Technology, Lanzhou 730050, China
3.State Grid Gansu Electric Company, Lanzhou 730030, China

Online:2016-11-15 Published:2016-12-02

Hadoop云平台MapReduce模型优化研究

张红1，2，王晓明1，曹洁2，马彦宏3，郭义戎1，王慜1

1.兰州理工大学电气与信息工程学院，兰州 730050
2.兰州理工大学计算机与通信学院，兰州 730050
3.国网甘肃省电力公司，兰州 730030

Abstract

Abstract: Sequential control of running mechanism of MapReduce model on Hadoop platform can lead to waste of computing resources. From the perspective of the fine-grained parallel data processing of each node, combined with multi-threads technique of Java shared memory, this paper optimizes MapReduce model and puts forward a MapReduce+OpenMP framework. This model is a distributed and parallel computing architecture based on Hadoop cloud platform, which combines computing resources of coarse and fine granularity. After programming and realizing on the GPS trajectory data of the taxi in the Hadoop distributed cluster environment, the results show that this distributed parallel computing model can really improve the computing efficiency of processing big data set, and it is an effective optimization and improvement to the MapReduce model of big data processing.

Key words: Hadoop, MapReduce, OpenMP, distributed, parallel

摘要： 针对Hadoop平台MapReduce分布式计算模型运行机制中的顺序制约而产生的计算资源浪费问题，从提高平台中每个执行节点的细粒度并行数据处理角度出发，结合Java共享内存多线程编程技术，对该模型进行了优化，提出一种MapReduce+OpenMP粗细粒度相结合的分布式并行计算模型。并在由四个节点组成的Hadoop集群环境下对不同规模大小的出租车GPS轨迹数据分析处理，验证该模型的性能和效率，实验结果证明MapReduce+OpenMP分布式并行计算模型确实能够提高针对大数据集的计算效率，是对Hadoop平台大数据分析处理模型有效的完善和优化。

关键词: Hadoop, MapReduce, OpenMP, 分布式, 并行

ZHANG Hong1，2, WANG Xiaoming1, CAO Jie2, MA Yanhong3, GUO Yirong1, WANG Min1. Research on optimized MapReduce model of Hadoop cloud platform[J]. Computer Engineering and Applications, 2016, 52(22): 22-25.

张红1，2，王晓明1，曹洁2，马彦宏3，郭义戎1，王慜1. Hadoop云平台MapReduce模型优化研究[J]. 计算机工程与应用, 2016, 52(22): 22-25.

[1]	LI Junli. Parallel Mutual-Information Computation of Categorical Data Based on Spark [J]. Computer Engineering and Applications, 2021, 57(7): 95-100.
[2]	SHI Jieyuan, YUAN Zhiyong, LIAO Xiangyun, ZHAO Jianhui. Multirate Systematic Framework for Magnetic Levitation Visuo-Haptic Interaction [J]. Computer Engineering and Applications, 2021, 57(5): 197-203.
[3]	TANG Rui, JIAO Jiye, XU Huahao. Design of Hardware Accelerator for Embedded Convolutional Neural Network [J]. Computer Engineering and Applications, 2021, 57(4): 252-257.
[4]	YANG Luyue, ZHANG Shumei, ZHAO Junli. Dynamic Expression Recognition with Partial Occlusion Based on Parallel Gan [J]. Computer Engineering and Applications, 2021, 57(24): 168-178.
[5]	ZHU Meng, MIN Weidong, ZHANG Yu, DUAN Jingwen. Parallel Selective Kernel Attention Based on HardSoftmax [J]. Computer Engineering and Applications, 2021, 57(21): 95-101.
[6]	JIANG Kui, QIU Yuandong, ZHENG Haocheng. ICMPv6 DDoS Attack Detection Method Based on Information Entropy and LSTM [J]. Computer Engineering and Applications, 2021, 57(21): 148-154.
[7]	WANG Hairui, XIANYU Jianchuan. Application of Distributed Generation Configuration Based on Improved Sparrow Search Algorithm [J]. Computer Engineering and Applications, 2021, 57(20): 245-252.
[8]	TIAN Yang, CHEN Zhigang, SONG Xinxia, LI Tianming. Overview of Blockchain Application in Supply Chain Management [J]. Computer Engineering and Applications, 2021, 57(19): 70-83.
[9]	ZHANG Jun, LIAO Xuehua, YU Xuling, LEI Meng. Research on Realizing Relational Database In-Memory Storage Model [J]. Computer Engineering and Applications, 2021, 57(19): 123-128.
[10]	FENG Kai, LI Jing. Subnetwork Reliability of k-Ary n-Cube Networks [J]. Computer Engineering and Applications, 2021, 57(16): 83-89.
[11]	WU Dongyang, DOU Jianping, LI Jun. Design of Digital Twin System for Quadrotor [J]. Computer Engineering and Applications, 2021, 57(16): 237-244.
[12]	LI Leixiao, DENG Dan, LI Jie, WANG Yongsheng. All-to-All Comparison Computing Data Distribution Strategy Based on Particle Swarm Optimization [J]. Computer Engineering and Applications, 2021, 57(15): 109-117.
[13]	XIANG Yixuan, JIANG He, PAN Pinchen, SUN Conghui. Study on [K]-means Clustering Algorithm of Quadratic Power Coupling [J]. Computer Engineering and Applications, 2021, 57(14): 95-102.
[14]	LI Jian, ZHANG Dawei, JIANG Xiaoming, XIANG Liyun. Review on Parallelized Flood Inundation Models [J]. Computer Engineering and Applications, 2021, 57(13): 1-7.
[15]	SUN Ming, CHEN Xin. Design Method of Convolutional Neural Network Accelerator [J]. Computer Engineering and Applications, 2021, 57(13): 77-84.

Research on optimized MapReduce model of Hadoop cloud platform

Hadoop云平台MapReduce模型优化研究

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics