Review of Classification Methods for Unbalanced Data Sets

doi:10.3778/j.issn.1002-8331.2107-0097

Abstract

Abstract:

The characteristics of unbalanced data sets lead to many difficult problems in classification. The classification methods of unbalanced data sets are analyzed and summarized. Firstly, the classification methods of unbalanced data sets are introduced from three perspectives of under-sampling, over-sampling and mixed sampling in detail. In the under-sampling method, it is divided into three technical methods based on [K]-Nearest Neighbor[(KNN)], Bagging and Boosting. In the over-sampling method, the classification method is analyzed from the perspectives of Synthetic Minority Over-sampling Technology（SMOTE） and Support Vector Machine（SVM）. The advantages and disadvantages of the algorithm are compared, and the performance of the algorithm is analyzed and summarized under the same data sets. Then, the classification methods of unbalanced data sets are summarized from four aspects：deep learning, extreme learning machine, cost sensitivity and feature selection. Finally, the future work direction is prospected.

Key words: unbalanced data set, classification, sampling method, [K]-Nearest Neighbor[(KNN)], Synthetic Minority Over-sampling Technology（SMOTE）, deep learning

摘要：

不平衡数据集的特点导致了在分类时产生了诸多难题。对不平衡数据集的分类方法进行了分析与总结。在数据采样方法中从欠采样、过采样和混合采样三方面介绍不平衡数据集的分类方法；在欠采样方法中分为基于[K]近邻、Bagging和Boosting三种方法；在过采样方法中从合成少数过采样技术（Synthetic Minority Over-sampling Technology，SMOTE）、支持向量机（Support Vector Machine，SVM）两个角度来分析不平衡数据集的分类方法；对这两类采样方法的优缺点进行了比较，在相同数据集下比较算法的性能并进行分析与总结。从深度学习、极限学习机、代价敏感和特征选择四方面对不平衡数据集的分类方法进行了归纳。最后对下一步工作方向进行了展望。

关键词: 不平衡数据集, 分类, 采样方法, [K]近邻[(KNN)], 合成少数过采样技术（SMOTE）, 深度学习

WANG Le, HAN Meng, LI Xiaojuan, ZHANG Ni, CHENG Haodong. Review of Classification Methods for Unbalanced Data Sets[J]. Computer Engineering and Applications, 2021, 57(22): 42-52.

王乐，韩萌，李小娟，张妮，程浩东. 不平衡数据集分类方法综述[J]. 计算机工程与应用, 2021, 57(22): 42-52.

References

[1] PHUA C，ALAHAKOON D，LEE V.Minority report in fraud detection：classification of skewed data[J].ACM SIGKDD Explorations Newsletter，2004，6（1）：50-59.
[2] KRAWCZYK B，GALAR M，JELE N L，et al.Evolutionary under-sampling boosting for imbalanced classification of breast cancer malignancy[J].Applied Soft Computing，2016，38：714-726.
[3] ALQATAWNA J，FARIS H，JARADAT K，et al.Improving knowledge based spam detection methods：the effect of malicious related features in imbalance data distribution[J].International Journal of Communications Network & System Sciences，2015，8（5）：118-129.
[4] 刘定祥，乔少杰，张永清，等.不平衡分类的数据采样方法综述[J].重庆理工大学学报（自然科学），2019，33（7）：102-112.
LIU D X，QIAO S J，ZHANG Y Q，et al.A survey on data sampling methods in imbalance classification[J].Journal of Chongqing University of Technology（Natural Science），2019，33（7）：102-112.
[5] 李勇，刘战东，张海军.不平衡数据的集成分类算法综述[J].计算机应用研究，2014，31（5）：1287-1291.
LI Y，LIU Z D，ZHANG H J.Review on ensemble algorithms for imbalanced data classification[J].Application Research of Computers，2014，31（5）：1287-1291.
[6] 杨明，尹军梅，吉根林.不平衡数据分类方法综述[J].南京师范大学学报（工程技术版），2008，8（4）：7-12.
YANG M，YIN J M，JI G L.Classification methods on imbalanced data：a survey[J].Journal of Nanjing Normal University（Engineering and Technology Edition），2008，8（4）：7-12.
[7] PRUSA J，KHOSHGOFTAAR T M，DITTMAN D J，et al.Using random under-sampling to alleviate class imbalance on tweet sentiment data[C]//Proceedings of the 16th IEEE International Conference on Information Reuse and Integration，Aug 13，2015，Piscataway：IEEE，2015：197-202.
[8] BATISTA G E，PRATI R C，MONARD M C.A study of the behavior of several methods for balancing machine learning training data[J].ACM SIGKDD Explorations Newsletter，2004，6（1）：20-29.
[9] KANG Q，CHEN X S，LI S S，et al.A noise-filtered under-sampling scheme for imbalance classification[J].IEEE Transactions on Cybernetics，2017，4：4263-4274.
[10] HOU Y，LI L，LI B，et al.An anti-noise ensemble algorithm for imbalance classification[J].Intelligent Data Analysis，2019，23（6）：1205-1217.
[11] BARANDELA R，SáNCHEZ J S，VALDOVINOS R M.New applications of ensembles of classifiers[J].Pattern Analysis & Applications，2003，6（3）：245-256.
[12] HIDO S，KASHIMA H，TAKAHASHI Y.Roughly balanced bagging for imbalanced data[J].Statal Analysis & Data Mining，2010，2（5/6）：412-426.
[13] LI C.Classifying imbalanced data using a bagging ensemble variation（BEV）[C]//Proceedings of the 45th Annual Southeast Regional Conference on ACM-SE，Mar 23，2017：203-208.
[14] SEIFFERT C，KHOSHGOFTAAR T，HULSE J V，et al.RUSBoost：a hybrid approach to alleviating class imbalance[J].IEEE Transactions on Systems，Man and Cybernetics，2010，40（1）：185-197.
[15] FREUND Y，SCHAPIRE R E.A decision-theoretic generalization of on-line learning and an application to boosting[J].Journal of Computer and System Sciences，1997，55（1）：119-139.
[16] BROWN G，WYATT J，HARRIS R，et al.Diversity crea-
tion methods：a survey and categorisation[J].Information Fusion，2005，6（1）：5-20.
[17] GALAR M，FERNáNDEZ A，BARRENECHEA E，et al.EUSBoost：enhancing ensembles for highly imbalanced data-sets by evolutionary under-sampling[J].Pattern Recog-
nition，2013，46：3460-3471.
[18] GONG J，KIM H.RHSBoost：improving classifification performance in imbalance data[J].Computational Statistics and Data Analysis，2017，111：1-13.
[19] LIN W C，TSAI C F，HU Y H，et al.Clustering-based under-sampling in class-imbalanced data[J].Information Sciences，2017，409/410：17-26.
[20] LIU X Y，WU J，ZHOU Z H.Exploratory under-sampling for class-imbalance learning[J].IEEE Transactions on Systems，Man and Cybernetics，2009，39（2）：539-550.
[21] 崔彩霞，曹付元，梁吉业.基于密度峰值聚类的自适应欠采样方法[J].模式识别与人工智能，2020，33（9）：811-819.
CUI C X，CAO F Y，LIANG J Y.Adaptive undersampling based on density peak clustering[J].Pattern Recognition and Artificial Intelligence，2020，33（9）：811-819.
[22] CHAWLA N V，BOWYER K，HALL L，et al.SMOTE：synthetic minority over-sampling technique[J].Journal of Artificial Intelligence Research，2002，16（1）：321-357.
[23] HAN H，WANG W Y，MAO B H.Borderline-SMOTE：a new over-sampling method in imbalanced data sets learning[C]//Proceedings of the 2005 International Conference on Intelligent Computing，Aug 23-26，2005.Berlin，Heidelberg：Springer，2005：878-887.
[24] BUNKHUMPORNPAT C，SINAPIROMSARAN K，LURSINSAP C.Safe-level-SMOTE：safe-level-synthetic minority over-sampling technique for handling the class imbalanced problem[C]//Proceedings of the 2009 Pacific-Asia Conference on Knowledge Discovery and Data Mining，Apr 19，2009.Berlin，Heidelberg：Springer，2009：475-482.
[25] BUNKHUMPORNPAT C，SINAPIROMSARAN K，LURSINSAP C.DBSMOTE：density-based synthetic minority over-sampling technique[J].Applied Intelligence，2012，36（3）：664-684.
[26] WANG S，YAO?X.Diversity?analysis?on?imbalanced datasets?by using ensemble models[C]//Proceedings of the 2009 IEEE Symposium on Computational Intelligence and Data Mining，Mar 2-Apr 30，2009.Piscataway：IEEE，2009：324-331.
[27] CHAWLA N V，LAZAREVIC A，HALL L O，et al.SMOTEBoost：improving prediction of the minority class in boosting[C]//Proceedings of the 2003 Knowledge Discovery in Databases，Jan 2003.Berlin，Heidelberg：Springer，2003：107-119.
[28] CORTES C，VAPNIK V N.Support-vector networks[J].Machine Learning，1995，20（3）：273-297.
[29] 黄海松，魏建安，康佩栋.基于不平衡数据样本特性的新型过采样SVM分类算法[J].控制与决策，2018，33（9）：1549-1558.
HUANG H S，WEI J A，KANG P D.New?over-sampling SVM classification algorithm based on unbalanced data sample characteristics[J].Control and Decision，2018，33（9）：1549-1558.
[30] ZHANG L M，TAN L M，LIU T S，et al.Classification study for the imbalanced data based on biased-SVM and the modified over-sampling?algorithm[J].Journal of Physics：Conference Series，2019，1237（2）：1-5.
[31] MA T，HOU Y，CHENG J J，et al.A novel method combining fuzzy SVM and sampling for imbalanced classification[J].International Journal of Applied Systemic Studies，2018，8（1）：1-31.
[32] TAO X，LI Q，REN C.Real-value negative selection over-sampling for imbalanced data set learning[J].Expert Systems with Applications，2019，129：118-134.
[33] TAO X，LI Q，GUO W.Adaptive weighted over-sampling for imbalanced datasets based on density peaks clustering with heuristic filtering[J].Information Sciences，2020，519：43-73.
[34] ESTABROOKS A，JO T，JAPKOWICZ N.A multiple re-sampling method for learning from imbalanced data sets[J].Computational Intelligence，2010，20（1）：18-36.
[35] SEIFFERT C，KHOSHGOFTAAR T M，VAN H J.Hybrid sampling for imbalanced data[J].Integrated Computer Aided Engineering，2009，16（3）：193-210.
[36] ESTABROOKS A，JAPKOWICZ N.A mixture-of-experts framework for learning from imbalanced data sets[C]//Proceedings of the 4th International Conference on Advances in Intelligent Data Analysis，Sep 3，2001.Berlin，Heidelberg：Springer，2001：34-43.
[37] 郑建华，刘双印，贺超波，等.基于混合采样策略的改进随机森林不平衡数据分类算法[J].重庆理工大学学报（自然科学版），2019，33（7）：113-123.
ZHENG J H，LIU S Y，HE C B，et al.Improved random forest classification algorithm for imbalance data based on hybrid sampling strategy[J].Journal of Chongqing University of Technology（Natural Science），2019，33（7）：113-123.
[38] 杨霞霞，苏锋，黄戌霞.基于改进随机森林算法的不平衡数据分类方法研究[J].网络安全技术与应用，2020（10）：70-71.
YANG X X，SU F，HUANG X X.Research on unbalanced data classification method based on improved random forest algorithm[J].Network Security Technology and Application，2020（10）：70-71.
[39] 周霄晗.基于深度神经网络对抗与集成的不平衡数据分类方法研究[D].杭州：浙江工业大学，2019.
ZHOU X H.Research on imbalanced data classification method based on deep neural network adversarial and ensembles[D].Hangzhou：Zhejiang University of Technology，2019.
[40] 解晓波.基于生成对抗网络的非均衡数据分类研究[D].南京：南京邮电大学，2020.
XIE X B.Research on classification of imbalanced dataset based on generative adversarial networks[D].Nanjing：Nanjing University of Posts and Telecommunications，2020.
[41] KONNO T，IWAZUME M.Pseudo-feature generation for imbalanced data analysis in deep learning[J].arXiv：1807.
06538，2018.
[42] 陈志，郭武.不平衡训练数据下的基于深度学习的文本分类[J].小型微型计算机系统，2020，41（1）：1-5.
CHEN Z，GUO W.Text classification based on depth learning on unbalanced data[J].Journal of Chinese Computer Systems，2020，41（1）：1-5.
[43] 李晗缦，王丽丹，段书凯.改进的超限学习机及其在不平衡数据中的应用[J].西南大学学报（自然科学版），2020，42（6）：140-148.
LI H M，WANG L D，DUAN S K.An improved extreme learning machine and its application in imbalance data[J].Journal of Southwest University（Natural Science Edition），2020，42（6）：140-148.
[44] 孙庆帅.基于不平衡数据分布的加权极限学习机算法研究[D].湘潭：湘潭大学，2019.
SUN Q S.Research on weighted extreme learning machine algorithm based on imbalanced data distribution[D].Xiangtan：Xiangtan University，2019.
[45] LI F L，ZHANG X Y，ZHANG X Q，et al.Cost-sensitive and hybrid-attribute measure multi-decision tree over imbalanced data sets[J].Information Sciences，2018，422：242-256.
[46] VEROPOULOS K，CAMPBELL C，CRISTIANINI N.Controlling the sensitivity of support vector machines[C]//Proceedings of the 1999 International Joint Conference on Artificial Intelligence，1999：55-60.
[47] TAO X，LI Q，GUO W，et al.Self-adaptive cost weights-based support vector machine cost-sensitive ensemble for imbalanced data classification[J].Information Sciences，2019，487：31-56.
[48] 平瑞，周水生，李冬.高度不平衡数据的代价敏感随机森林分类算法[J].模式识别与人工智能，2020，33（3）：249-257.
PING R，ZHOU S S，LI D.Cost sensitive random forest classification algorithm for highly unbalanced data[J].Pattern Recognition and Artificial Intelligence，2020，33（3）：249-257.
[49] ELKAN C.The foundations of cost-sensitive learning[C]//International Joint Conference on Artificial Intelligence，2001：973-978.
[50] CUI Y J，DAVIS S，CHENG C K，et al.A study of sample size with neural network[C]//Proceedings of the 3rd International Conference on Machine Learning & Cybernetics，Aug 26-29，2004.Piscataway：IEEE，2004：3444-3448.
[51] CHEN J，LIU S，LIU W，et al.A two-stage data preprocessing approach for software fault prediction[C]//Proceedings of the 8th International Conference on Software Security & Reliability，Jun 30-Jul 2，2014.Piscataway：IEEE，2014：20-29.
[52] PUTTEN P V D，SOMEREN M V.A bias-variance analysis of a real world learning problem：the CoIL challenge 2000[J].Machine Learning，2004，57（1/2）：177-195.
[53] NAM J，PAN S J，KIM S.Transfer defect learning[C]//Proceedings of the 35th International Conference on Software Engineering.Washington：IEEE Computer Society，2013：382-391.
[54] PAN S J，TSANG I W，KWOK J T，et al.Domain adaptation via transfer component analysis[J].IEEE Transactions on Neural Networks，2011，22（2）：199-210.
[55] 刘树毅，翟晔，刘东升.融合多策略特征筛选的跨项目软件缺陷预测[J].计算机工程与应用，2019，55（8）：53-58.
LIU S Y，ZHAI Y，LIU D S.Cross-project software defect prediction with multi-strategy feature filtering[J].Computer Engineering and Applications，2019，55（8）：53-58.
[56] GAO K，KHOSHGOFTAAR T M，WANG H，et al.Choosing software metrics for defect prediction：an investigation on feature selection techniques[J].Software：Practice and Experience，2011，41（5）：579-606.
[57] LARADJI I H，ALSHAYEB M，GHOUTI L.Software defect prediction using ensemble learning on selected features[J].Information & Software Technology，2015，58：388-402.