关键词: Hadoop, 小文件处理, Hadoop的分布式文件系统（HDFS）, MapReduce, 大数据

Abstract: Hadoop is designed to store and analyze large data, and it is good at processing large data sets. However, in practical applications, there are a large number of small files. There are four methods of handling massive small files generally which are default input format TextInputFormat, CombineFileInputFormat which is designed for handling small files, SequenceFile technology and Harballing technology. In order to compare the performance of these four technologies dealing with a large number of small files in the same Hadoop distributed environment, it uses a word frequency statistics program with typical data sets to compare the performance differences between the four small files processing technology. Experimental studies have shown that, when dealing with a large number of small files in different needs, choosing the appropriate handling method can improve the processing efficiency of a large number of small files to a large extent.

Key words: Hadoop, small files handling, Hadoop Distributed File System（HDFS）, MapReduce, big data