Computer Engineering and Applications ›› 2014, Vol. 50 ›› Issue (20): 115-118.
Previous Articles Next Articles
WAN Wan, ZHOU Guoxiang
Online:
Published:
宛 婉,周国祥
Abstract: In today’s “information explosion” society, data mining, because of mass data, faces a new challenges. When data mining turns to cloud computing platform to realize parallel, the study of parallel data random sampling further reduces the size of the data size. This paper presents a mapreduce parallel sampling algorithm which not only can clean up dirty data but also achieves the goal of equal probability sampling. The algorithm just needs to scan processed data only one time. It runs this algorithm in the hadoop platform and compares its performance with common random sampling. As a result, this new algorithm obtains a very high time efficiency. It is a kind of effective method which lays a good foundation for doing research on sampling in future. It can also promote data mining in the condition of facing mass data.
Key words: cloud computing, hadoop, mapreduce, parallel computing, data mining, random sampling
摘要: 在“信息爆炸”的当今社会,海量数据对数据挖掘提出新的挑战。在数据挖掘转向云计算平台实现并行化的同时,研究并行化数据随机抽样进一步降低处理的数据规模。提出一种单次扫描即可实现清理脏数据并实现等概率抽样的mapreduce并行抽样算法。在hadoop平台上实现并与普通随机抽样方法进行比较,得出其时间效率非常高,是一种行之有效的方法。为以后数据挖掘中的抽样研究和推动数据挖掘在海量数据下的发展奠定良好基础。
关键词: 云计算, hadoop, mapreduce, 并行计算, 数据挖掘, 随机抽样
WAN Wan, ZHOU Guoxiang. Massive data parallel random sampling based on hadoop[J]. Computer Engineering and Applications, 2014, 50(20): 115-118.
宛 婉,周国祥. Hadoop平台的海量数据并行随机抽样[J]. 计算机工程与应用, 2014, 50(20): 115-118.
0 / Recommend
Add to citation manager EndNote|Ris|BibTeX
URL: http://cea.ceaj.org/EN/
http://cea.ceaj.org/EN/Y2014/V50/I20/115