Survey of Deep Clustering Algorithm Based on Autoencoder

doi:10.3778/j.issn.1002-8331.2204-0049

Abstract

Abstract: As a common analysis method, cluster analysis is widely used in various scenarios. With the development of machine learning technology, deep clustering algorithm has also become a hot research topic, and the deep clustering algorithm based on autoencoder is one of the representative algorithms. To keep abreast of the development of deep clustering algorithms based on autoencoders, four models of autoencoders are introduced, and the representative algorithms in recent years are classified according to the structure of autoencoders. For the traditional clustering algorithm and the deep clustering algorithm based on autoencoder, experiments are compared and analyzed on the MNIST, USPS, Fashion-MNIST datasets. At last, the current problems of deep clustering algorithms based on autoencoders are summarized, and the possible research directions of deep clustering algorithms are prospected.

Key words: clustering algorithm, deep clustering, autoencoder, feature extraction

摘要： 聚类分析作为一种常见的分析方法，广泛应用于各种场景。随着机器学习技术的发展，深度聚类算法也成了当下研究的热点，基于自编码器的深度聚类算法是其中的代表算法。为了及时了解掌握基于自编码器的深度聚类算法的发展，介绍了四种自编码器的模型，对近些年代表性的算法依照自编码器的结构进行了分类。在MNIST、USPS、Fashion-MNIST数据集上，针对传统聚类算法和基于自编码器的深度聚类算法进行了实验对比、分析，最后对基于自编码器的深度聚类算法目前存在的问题进行了总结，展望了深度聚类算法的研究方向。

关键词: 聚类算法, 深度聚类, 自编码器, 特征提取

TAO Wenbin, QIAN Yurong, ZHANG Yiyang, MA Hengzhi, LENG Hongyong, MA Mengnan. Survey of Deep Clustering Algorithm Based on Autoencoder[J]. Computer Engineering and Applications, 2022, 58(18): 16-25.

陶文彬, 钱育蓉, 张伊扬, 马恒志, 冷洪勇, 马梦楠. 基于自编码器的深度聚类算法综述[J]. 计算机工程与应用, 2022, 58(18): 16-25.

References

[1] JAIN A K.Data clustering：50 years beyond K-means[J].Pattern Recognition Letters，2010，31（8）：651-666.
[2] 杨俊闯，赵超.K-Means聚类算法研究综述[J].计算机工程与应用，2019，55（23）：7-14.
YANG J C，ZHAO C.Survey on K-means clustering algorithm[J].Computer Engineering and Applications，2019，55（23）：7-14.
[3] ZHOU Y，WU H，LUO Q，et al.Automatic data clustering using nature-inspired symbiotic organism search algorithm[J].Knowledge-Based Systems，2019，163：546-557.
[4] ABUALIGAH L M，KHADER A T，HANANDEH E S.A new feature selection method to improve the document clustering using particle swarm optimization algorithm[J].Journal of Computational Science，2018，25：456-466.
[5] YU S S，CHU S W，WANG C M，et al.Two improved k-means algorithms[J].Applied Soft Computing，2018，68：747-755.
[6] XU J，LANGE K.Power [k]-means clustering[C]//Proceedings of the 36th International Conference on Machine Learning，2019：6921-6931.
[7] HERSHEY J R，CHEN Z，ROUX J L，et al.Deep clustering：discriminative embeddings for segmentation and separation[C]//2016 IEEE International Conference on Acoustics，Speech and Signal Processing（ICASSP），2016：31-35.
[8] CARON M，BOJANOWSKI P，JOULIN A，et al.Deep clustering for unsupervised learning of visual features[C]//European Conference on Computer Vision，2018.
[9] TSCHANNEN M，BACHEM O，LUCIC M.Recent advances in autoencoder-based representation learning[J].arXiv：1812.
905069，2018.
[10] XIE J，GIRSHICK R，FARHADI A.Unsupervised deep embedding for clustering analysis[C]//International Conference on Machine Learning，2016：478-487.
[11] GUO X，GAO L，LIU X，et al.Improved deep embedded clustering with local structure preservation[C]//International Joint Conferences on Artificial Intelligence Organization，2017：1753-1759.
[12] GUO X，LIU X，ZHU E，et al.Deep clustering with convolutional autoencoders[C]//International Conference on Neural Information Processing，2017：373-382.
[13] CAI J，WANG S，GUO W.Unsupervised embedded feature learning for deep clustering with stacked sparse auto-encoder[J].Expert Systems with Applications，2021，186：115729.
[14] JIANG Z，ZHENG Y，TAN H，et al.Variational deep embedding：an unsupervised and generative approach to clustering[J].arXiv：1611.05148，2016.
[15] ZHANG D，SUN Y，ERIKSSON B，et al.Deep unsupervised clustering using mixture of autoencoders[J].arXiv：1712.07788，2017.
[16] MIN E，GUO X，LIU Q，et al.A survey of clustering with deep learning：from the perspective of network architecture[J].IEEE Access，2018，6：39501-39514.
[17] MACQUEEN J.Some methods for classification and analysis of multivariate observations[C]//Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability，1967：281-297.
[18] YANG B，FU X，SIDIROPOULOS N D，et al.Towards K-means-friendly spaces：simultaneous deep learning and clustering[C]//Proceedings of the 34th International Conference on Machine Learning，2017：3861-3870.
[19] XIE J，GIRSHICK R，FARHADI A.Unsupervised deep embedding for clustering analysis[J].arXiv：1511.06335，2015.
[20] BEEFERMAN D，BERGER A.Agglomerative clustering of a search engine query log[C]//Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining，2000：407-416.
[21] PEARSON K.On lines and planes of closest fit to systems of points in space[J].The London，Edinburgh，and Dublin Philosophical Magazine and Journal of Science，1901，2（11）：559-572.
[22] BURGES C J C.A tutorial on support vector machines for pattern recognition[J].Data Mining and Knowledge Discovery，1998，2（2）：121-167.
[23] TAX D M J，DUIN R P W.Support vector domain description[J].Pattern Recognition Letters，1999，20（11/13）：1191-1199.
[24] DONATH W E，HOFFMAN A J.Lower bounds for the partitioning of graphs[J].IBM Journal of Research and Development，1973，17（5）：420-425.
[25] ESTER M，KRIEGEL H P，SANDER J，et al.Density-based spatial clustering of applications with noise[C]//International Conference on Knowledge Discovery and Data Mining，1996：6.
[26] RUMELHART D E，HINTON G E，WILLIAMS R J.Learning representations by back-propagating errors[J].Nature，1986，323（6088）：533-536.
[27] BOURLARD H，KAMP Y.Auto-association by multilayer perceptrons and singular value decomposition[J].Biological Cybernetics，1988，59（4）：291-294.
[28] MASCI J，MEIER U，CIRE?AN D，et al.Stacked convolutional auto-encoders for hierarchical feature extraction[C]//Proceedings of the 21st International Conference on Artificial Neural Networks-Volume Part I，2011：52-59.
[29] NG A.Sparse autoencoder[J].CS294A Lecture Notes，2011，72（2011）：1-19.
[30] KINGMA D P，WELLING M.Auto-encoding variational bayes[J].arXiv：1312.6114，2013.
[31] LAURENS V D M，HINTON G.Visualizing data using t-SNE[J].Journal of Machine Learning Research，2008，9（2605）：2579-2605.
[32] MRABAH N，KHAN N M，KSANTINI R，et al.Deep clustering with a dynamic autoencoder：from reconstruction towards centroids construction[J].Neural Networks，2020，130：206-228.
[33] DIZAJI K G，HERANDI A，HUANG H.Deep clustering via joint convolutional autoencoder embedding and relative entropy minimization[C]//International Conference on Neural Information Processing，2017：5736-5745.
[34] WICKRAMASINGHE C S，MARINO D L，MANIC M.ResNet autoencoders for unsupervised feature learning from high-dimensional data：deep models resistant to performance degradation[J].IEEE Access，2021，9：40511-40520.
[35] HUANG P，HUANG Y，WANG W，et al.Deep embedding network for clustering[C]//2014 22nd International Conference on Pattern Recognition，2014：1532-1537.
[36] BOTTOU L.Large-scale machine learning with stochastic gradient descent[J].Physica-Verlag HD，2010.
[37] DILOKTHANAKUL N，MEDIANO P A M，GARNELO M，et al.Deep unsupervised clustering with Gaussian mixture variational autoencoders[J].arXiv：1611.02648，2016.
[38] KINGMA D P，SALIMANS T，JOZEFOWICZ R，et al.Improved variational inference with inverse autoregressive flow[J].Advances in Neural Information Processing Systems，2016，29：4743-4751.
[39] YANG X，DENG C，ZHENG F，et al.Deep spectral clustering using dual autoencoder network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：4066-4075.
[40] 陈锐，唐永强，张彩霞，等.深度多网络嵌入聚类[J].模式识别与人工智能，2021，34（1）：14-24.
CHEN R，TANG Y Q，ZHANG C X，et al.Deep multi-network embedded clustering[J].Pattern Recognition and Artificial Intelligence，2021，34（1）：14-24.
[41] DU G，ZHOU L，YANG Y，et al.Deep multiple auto-encoder-based multi-view clustering[J].Data Science and Engineering，2021，6（3）：323-338.
[42] MUNKRES J R.Algorithms for the assignment and transportation problems[J].Journal of The Society for Industrial and Applied Mathematics，1957，10：196-210.
[43] 向宽，李松松，栾明慧，等.基于改进Faster RCNN的铝材表面缺陷检测方法[J].仪器仪表学报，2021，42（1）：191-198.
XIANG K，LI S S，LUAN M H，et al.Aluminum product surface defect detection method based on improved faster RCNN[J].Chinese Journal of Scientific Instrument，2021，42（1）：191-198.
[44] 张晨阳，黄腾，吴壮壮.基于K-Means聚类与深度学习的RGB-D SLAM算法[J].计算机工程，2022，48（1）：236-244.
ZHANG C Y，HUANG T，WU Z Z.RGB-D SLAM algorithm based on K-means clustering and deep learning[J].Computer Engineering，2022，48（1）：236-244.
[45] 杨杰文，章光，陈西江，等.融合深度学习聚类分割和形态学的混凝土表面裂缝量化识别[J].激光与光电子学进展，2020，57（22）：242-252.
YANG J W，ZHANG G，CHEN X J，et al.Quantitative identification of concrete surface cracks based on deep learning clustering segmentation and morphology[J].Laser & Optoelectronics Progress，2020，57（22）：242-252.
[46] 杨焰飞，曹阳.基于深度学习的无人机拍摄图像绝缘子目标检测[J].激光杂志，2020，41（10）：63-66.
YANG Y F，CAO Y.Image insulator target detection based on deep learning for UAV[J].Laser Journal，2020，41（10）：63-66.
[47] XIE Y，LIN B，QU Y，et al.Joint deep multi-view learning for image clustering[J].IEEE Transactions on Knowledge and Data Engineering，2021，33（11）：3594-3606.
[48] 高蕴梅.MGTL-SAE精细特征学习的图像资源快速检索[J].情报资料工作，2020，41（5）：79-87.
GAO Y M.Quick retrieval of image resources based on MGTL-SAE fine feature learning[J].Information and Documentation Services，2020，41（5）：79-87.
[49] 李艳红，赵宏伟，王素格，等.面向微博文本流的负面情感突发话题检测[J].计算机应用，2020，40（12）：3458-3464.
LI Y H，ZHAO H W，WANG S G，et al.Detection of negative emotion burst topic in microblog text stream[J].Journal of Computer Applications，2020，40（12）：3458-3464.
[50] 陈珂，叶颖雅，马乙平，等.用于微博情感分析的深度学习网络模型[J].计算机与数字工程，2020，48（7）：1674-1681.
CHEN K，YE Y Y，MA Y P，et al.Deep learning network model for micro-blog sentiment analysis[J].Computer & Digital Engineering，2020，48（7）：1674-1681.
[51] 桑凯恒，张繁昌，李传辉.地震倒谱特征参数谱聚类地震相分析方法[J].石油地球物理勘探，2021，56（1）：38-48.
SANG K H，ZHANG F C，LI C H.Seismic cepstrum characteristic parameter spectral clustering seismic facies analysis method[J].Oil Geophysical Prospecting，2021，56（1）：38-48.
[52] 李亚芳，梁烨，冯韦玮，等.基于社区优化的深度网络嵌入方法[J].计算机应用，2021，41（7）：1956-1963.
LI Y F，LIANG Y，FENG W Y，et al.Deep network embedding method based on community optimization[J].Journal of Computer Applications，2021，41（7）：1956-1963.
[53] 邱亚星，王希栋，边森，等.基于聚类分析和深度学习的多频多模网络负载均衡优化[J].电信科学，2020，36（7）：156-162.
QIU Y X，WANG X D，BIAN S，et al.Load balancing based on clustering analysis and deep learning for multi-frequency and multi-mode network[J].Telecommunications Science，2020，36（7）：156-162.
[54] LIU Z，LI J G，SHEN Z Q，et al.Learning efficient convolutional networks through network slimming[C]//2017 IEEE International Conference on Computer Vision，2017.
[55] XIANG Q，WANG X，SONG Y，et al.One-dimensional convolutional neural networks for high-resolution range profile recognition via adaptively feature recalibrating and automatically channel pruning[J].International Journal of Intelligent Systems，2021，36（1）：332-361.
[56] 王晓丹，来杰，李睿，等.多层去噪极限学习机[J].吉林大学学报（工学版），2020，50（3）：1031-1039.
WANG X D，LAI J，LI R，et al.Multilayer denoising extreme learning machine[J].Journal of Jilin University（Engineering and Technology Edition），2020，50（3）：1031-1039.
[57] LI S，KAWALE J，FU Y.Deep collaborative filtering via marginalized denoising auto-encoder[C]//Proceedings of the 24th ACM International on Conference on Information and Knowledge Management，2015：811-820.