New approach for clustering similar duplicate records based on high dimensions

doi:10.3778/j.issn.1002-8331.2008.29.048

Computer Engineering and Applications ›› 2008, Vol. 44 ›› Issue (29): 171-173.DOI: 10.3778/j.issn.1002-8331.2008.29.048

• 数据库、信号与信息处理 • Previous Articles Next Articles

New approach for clustering similar duplicate records based on high dimensions

CAO Qu-jiang,DONG Ming

Department of Computer and Electrical Engineering，University of Shanghai for Science and Technology，Shanghai 200093，China

Received:2007-12-03 Revised:2008-03-03 Online:2008-10-11 Published:2008-10-11
Contact: CAO Qu-jiang

一种在高维空间中聚类检测重复记录的新方法

曹渠江,董明

上海理工大学计算机与电气工程学院，上海 200093

通讯作者: 曹渠江

Abstract

Abstract: Data cleaning is an important area of data warehouse.Detecting duplicate records is a critical task in data cleaning.A new duplicate detection methods is proposed in this paper.The approach based on N-gram mappings all records in a relation to a high dimensions and clusters duplicate records through an improved DBSCAN algorithms which named IDS.IDS can cluster approximately duplicate records by using adjustable density.At last the experimental results prove the approach’s effectiveness.

Key words: approximately duplicate database, N-gram, Intrusion Detection System（IDS）

摘要： 数据清理是构建数据仓库中的一个重要研究领域。检测相似重复记录是数据清洗中一项非常重要的任务。提出了一种聚类检测相似重复记录的新方法，该方法是基于N-gram将关系表中的记录映射到高维空间中，并且通过可调密度的改进型DBSCAN算法IDS来聚类检测相似重复记录。并用实验证明了这种方法的有效性。

关键词: 相似重复记录, N-gram, 入侵检测系统

CAO Qu-jiang,DONG Ming. New approach for clustering similar duplicate records based on high dimensions[J]. Computer Engineering and Applications, 2008, 44(29): 171-173.

曹渠江,董明. 一种在高维空间中聚类检测重复记录的新方法[J]. 计算机工程与应用, 2008, 44(29): 171-173.

[1]	WANG Xiuzhen, CONG Rui, WANG Fei. Novel spelling correction algorithm for online query [J]. Computer Engineering and Applications, 2015, 51(14): 113-119.
[2]	ZHANG Wei1, DONG Qunfeng2. Design and implementation of cloud security comprehensive analysis system [J]. Computer Engineering and Applications, 2014, 50(19): 89-94.
[3]	HU Xiaopeng, YUAN Qi, GENG Xinhui, ZHU Shu. Building and profiling Chinese-English 3-tuple comparable corpora [J]. Computer Engineering and Applications, 2014, 50(13): 153-157.
[4]	NIJAT Najmidin1，2, MAHMUD Mamat3, TURGUN Ibrahim4. Experimental study of N-gram based Uyghur part of speech tagging [J]. Computer Engineering and Applications, 2012, 48(25): 137-140.
[5]	YUAN Qi, XIAO Jian, SONG Jinping, ZHU Shu, WAN Ying, XU Liang. Research and application of automatic profiling technologies based on 3-tuple comparable corpora [J]. Computer Engineering and Applications, 2012, 48(16): 129-132.
[6]	FENG Jing-hua，Gulila·Altenbek，Mayra·Hapar. Kazakh organization name recognition based on N-gram model [J]. Computer Engineering and Applications, 2010, 46(31): 135-138.
[7]	OU Jian-lin，LIN Qian，SHI Xiao-dong. Application of latent semantic analysis in continuous speech recognition [J]. Computer Engineering and Applications, 2009, 45(32): 111-113.
[8]	SHI Guang^1,3,HAO Ying²,LU Xiu-lin⁴,LIU Quan-fu⁴,XU Long-wei⁴,LEI Yu-guo⁴. Solution to getting inter-independent characters used to network intrusion detection system [J]. Computer Engineering and Applications, 2009, 45(2): 125-127.
[9]	LIU Feng-chen¹,LIU Qing-wen²,HU Yue²,HUANG He¹. Space and time optimized algorithm of n-Gram/2L index structure [J]. Computer Engineering and Applications, 2008, 44(5): 180-183.
[10]	YANG Chun,YANG Hai-dong,DENG Fei-qi. Application of active learning algorithms to intrusion detection of mobile Ad-Hoc Networks [J]. Computer Engineering and Applications, 2007, 43(31): 16-18.

New approach for clustering similar duplicate records based on high dimensions

一种在高维空间中聚类检测重复记录的新方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 10

Recommended Articles

Metrics