Computer Engineering and Applications ›› 2014, Vol. 50 ›› Issue (20): 115-118.

Previous Articles     Next Articles

Massive data parallel random sampling based on hadoop

WAN Wan, ZHOU Guoxiang   

  1. School of Computer and Information, Hefei University of Technology, Hefei 230009, China
  • Online:2014-10-15 Published:2014-10-28

Hadoop平台的海量数据并行随机抽样

宛  婉,周国祥   

  1. 合肥工业大学 计算机与信息学院,合肥 230009

Abstract: In today’s “information explosion” society, data mining, because of mass data, faces a new challenges. When data mining turns to cloud computing platform to realize parallel, the study of parallel data random sampling further reduces the size of the data size. This paper presents a mapreduce parallel sampling algorithm which not only can clean up dirty data but also achieves the goal of equal probability sampling. The algorithm just needs to scan processed data only one time. It runs this algorithm in the hadoop platform and compares its performance with common random sampling. As a result, this new algorithm obtains a very high time efficiency. It is a kind of effective method which lays a good foundation for doing research on sampling in future. It can also promote data mining in the condition of facing mass data.

Key words: cloud computing, hadoop, mapreduce, parallel computing, data mining, random sampling

摘要: 在“信息爆炸”的当今社会,海量数据对数据挖掘提出新的挑战。在数据挖掘转向云计算平台实现并行化的同时,研究并行化数据随机抽样进一步降低处理的数据规模。提出一种单次扫描即可实现清理脏数据并实现等概率抽样的mapreduce并行抽样算法。在hadoop平台上实现并与普通随机抽样方法进行比较,得出其时间效率非常高,是一种行之有效的方法。为以后数据挖掘中的抽样研究和推动数据挖掘在海量数据下的发展奠定良好基础。

关键词: 云计算, hadoop, mapreduce, 并行计算, 数据挖掘, 随机抽样