Optimization of MapReduce data flow with message broker mechanism

Computer Engineering and Applications ›› 2013, Vol. 49 ›› Issue (5): 120-122.

Previous Articles Next Articles

Optimization of MapReduce data flow with message broker mechanism

GE Junwei1, JIANG Xian2, FANG Yiqiu2

1.Department of Library, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
2.College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China

Online:2013-03-01 Published:2013-03-14

消息代理机制下的MapReduce数据流优化

葛君伟1，蒋仙2，方义秋2

1.重庆邮电大学图书馆，重庆 400065
2.重庆邮电大学计算机科学与技术学院，重庆 400065

Abstract

Abstract: MapReduce programming model is a kind of parallel computing framework which is distributed under the environment of mass data processing system. Currently, the MapReduce applications are widely used for commercial data intensive computing, the data transmission between the nodes on cluster has a large extent dependence. It causes that the load of message handling between the nodes is heavy. This paper puts forward an improved model of MapReduce based on message broker mechanism, to optimize the MapReduce data flow. The experimental data indicates that based on message broker mechanism the MapReduce framework can improve the load balance in data intensive applications.

Key words: message broker, MapReduce, data intensive computing, data flow

摘要： 摘要：MapReduce编程模型是广泛应用于云计算环境下处理海量数据的一种并行计算框架。然而该框架下的面向数据密集型计算，集群节点间的数据传输依赖性较强，造成节点间的消息处理负载过重。提出基于消息代理机制的MapReduce改进模型，优化数据流。经实验数据表明，基于消息代理机制的MapReduce框架能提高数据密集型应用上的负载均衡。

关键词: 消息代理, MapReduce, 数据密集型计算, 数据流

GE Junwei1, JIANG Xian2, FANG Yiqiu2. Optimization of MapReduce data flow with message broker mechanism[J]. Computer Engineering and Applications, 2013, 49(5): 120-122.

葛君伟1，蒋仙2，方义秋2. 消息代理机制下的MapReduce数据流优化[J]. 计算机工程与应用, 2013, 49(5): 120-122.

[1]	CHEN Yuanwen. Application of MapReduce Technology in Problem of Material Transportation and Stowage [J]. Computer Engineering and Applications, 2021, 57(12): 273-278.
[2]	LIU Jun, LI Wei, WU Mengting, CHEN Qifeng. New Design of Image Parallel Processing Model Based on Hadoop Platform [J]. Computer Engineering and Applications, 2019, 55(6): 186-190.
[3]	JI Changqing1，2, XIAO Peng3, LIU Chang4, WANG Zumin2, XI Fang2, SHAO Yinbo1, LI Zeyu2. Mobile Medical Call Algorithms Based on Spatial kNN Query [J]. Computer Engineering and Applications, 2019, 55(2): 206-212.
[4]	WANG Dezheng1, ZHANG Yinong1, YANG Fan2. Implementation of parallel PLS algorithm of process monitoring using MapReduce [J]. Computer Engineering and Applications, 2018, 54(24): 61-65.
[5]	YANG Rongze, LIU Yi. Abnormal data flow oriented selection integration method of multiple classifiers [J]. Computer Engineering and Applications, 2018, 54(2): 107-113.
[6]	CHEN Wanghu, YU Maoyi, MA Shengjun. Training BP neural networks with MapReduce based on sample data slice disruptions [J]. Computer Engineering and Applications, 2018, 54(2): 137-143.
[7]	XIA Xiaoyun, ZHANG Renbin, XIE Rui, WANG Cong. MapReduce approach for defect inspection of TFT-LCD [J]. Computer Engineering and Applications, 2017, 53(5): 202-206.
[8]	WANG Yonggui1, ZHANG Yan1, YANG Dongdong2. Research on algorithm of community discovery of wireless city based on MapReduce [J]. Computer Engineering and Applications, 2017, 53(4): 106-112.
[9]	YANG Xiaochuan, JIANG Jun, MA Xiaodong, QI Fengbin. Program slicing technology based on critical variable dataflow analysis algorithm in GCC compiler [J]. Computer Engineering and Applications, 2017, 53(24): 40-47.
[10]	LUO Jun, LI Jinhua. LSHBMRPK-means algorithm and its application. Computer Engineering and Applications [J]. Computer Engineering and Applications, 2017, 53(21): 62-67.
[11]	CHEN Yanan, ZHU Xijun. Association analysis of TCM asthma medication combination based on Hadoop [J]. Computer Engineering and Applications, 2017, 53(13): 95-98.
[12]	LI Sanmiao, LI Longshu. Performance analysis of four methods for handling small files in Hadoop [J]. Computer Engineering and Applications, 2016, 52(9): 44-49.
[13]	XIONG Zheng1，2, WANG Jinming2, ZHENG Haiyan1，2, LI Kunming1, XU Lizhen2, CHONG Zhihong2. RDF pattern matching algorithm with sorted view [J]. Computer Engineering and Applications, 2016, 52(8): 62-69.
[14]	WANG Yonggui, WU Chao, DAI Wei. K-means algorithm of random sample based on MapReduce [J]. Computer Engineering and Applications, 2016, 52(8): 74-79.
[15]	ZHANG Hong1，2, WANG Xiaoming1, CAO Jie2, MA Yanhong3, GUO Yirong1, WANG Min1. Research on optimized MapReduce model of Hadoop cloud platform [J]. Computer Engineering and Applications, 2016, 52(22): 22-25.

Optimization of MapReduce data flow with message broker mechanism

消息代理机制下的MapReduce数据流优化

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics