关键词: 异常检测, 多分辨率网格, 高维数据, 子空间, 数据挖掘

Abstract:

As an important means of data mining, anomaly detection is widely used in the field of data analysis. However, existing anomaly detection algorithms often need to adjust different parameters for different data to achieve the corresponding detection effect. In the face of big data, the detection time efficiency of existing algorithms is not satisfactory. The anomaly detection technology based on grid can well solve the problem of time efficiency of low-dimensional data anomaly detection. However, the detection accuracy depends heavily on the grid partition scale and density threshold parameters, which have poor robustness and cannot be well extended to different types of data sets. Based on the above problems, the proposed method firstly introduces a submatrix partition parameter with good robustness, divides high-dimensional data into several low-dimensional subspaces, and makes the anomaly detection algorithm carry out on the subspaces, so as to ensure the applicability of high-dimensional data. Then, an anomaly detection algorithm based on multi-resolution grid is proposed. Through the multi-resolution grid division from sparse to dense, the local anomaly factors of data points in different scale grids are comprehensively weighed, and the final output is the score ranking of global outliers. Experimental results show that the newly introduced submatrix partition parameters have good robustness, and the method can adapt to high-dimensional data well, and can get good detection effect on multiple public data sets, providing an efficient solution for solving the problems related to anomaly detection of high-dimensional data.

Key words: anomaly detection, multi-resolution grid, high dimensional data, subspace, data mining