关键词: 高通量测序, 重复数据删除, 近似去重, 布谷过滤器

Abstract: The next generation sequencing needs to be processed by cloud computing due to its large data volume, complex pipeline and high requirements of computing resources. Cloud computing approach necessitates that the sequencing data is uploaded to the cloud platform first. The randomness of the sequencing process results in great differences at the binary level even dealing with the same sample or two similar samples. Existing methods of deduplication do not effectively identify duplicate contents in such sequencing results. Uploading and storing these duplicate data not only consume network bandwidth, but also waste storage space. Aiming to the existing methods are based on the binary feature of file, not effectively use the similarity features of sequencing results, NPD（Near Probability Deduplication） method is proposed for massive high-throughput sequencing data oriented cloud platform. It uses SimHash to calculate the block fingerprints of sequence and quality information in FastQ file, and then the fingerprints are quickly detected by double cuckoo filter of client and cloud platform. At the final stage, the cloud platform uses approximate algorithm to near deduplicate fingerprints. Experimental results show that the NPD method can improve deduplication ratio, reduce network traffic, shorten data upload time, and support massive data processing, and has a good practical value.

Key words: high-throughput sequence data, data de-duplication, near de-duplication, cuckoo filter