基于对比学习的矢量化特征空间嵌入聚类

doi:10.3778/j.issn.1002-8331.2209-0338

摘要/Abstract

摘要： 深度嵌入聚类（deep embedding clustering, DEC）算法只通过自编码器，以单一实例重构的方式将数据嵌入到低维矢量化特征空间中进行聚类，而忽略了不同实例之间的关系，导致可能无法很好地区分嵌入空间中的实例。针对上述问题，提出基于对比学习的矢量化特征空间嵌入聚类（vectorized feature space embedded clustering based on contrastive learning, VECCL）方法。通过对比学习以辨识数据实例之间异同性的方式，从数据中提取出具有同近异远聚类语义的特征，并作为先验知识带入DEC中，引导自编码器初始化带有深层数据信息的低维聚类特征空间。同时利用软分类标签构造熵损失，与自编码器的重构损失一起作为正则化项引入聚类损失函数中，共同细化聚类。实验结果表明，所提方法提取特征的能力更强，与DEC方法在数据集CIFAR10、CIFAR100和STL10上的实验结果相比，ACC分别提升48.1个百分点、23.1个百分点和41.8个百分点，NMI分别提升41.0个百分点、25.2个百分点和39.0个百分点，ARI分别提升45.4个百分点、16.4个百分点和41.8个百分点。

关键词: 深度聚类, 对比学习, 自编码器, 矢量化特征空间, 嵌入聚类

Abstract: The deep embedding clustering (DEC) algorithm only embeds data into a low-dimensional vectorized feature space by autoencoder with a single instance reconstruction for clustering, and ignores the relationship between different instances, which leads to the instances in the embedding space may not be well distinguished from each other. To address the above problems, vectorized feature space embedded clustering based on contrastive learning (VECCL) method is proposed. By contrastive learning to identify the dissimilarity between data instances in a way, features with homogeneous near and different far clustering semantics are extracted from the data and brought into DEC as prior knowledge to guide the autoencoder to initialize a low-dimensional clustering feature space with deep data information. At the same time, the entropy loss constructed by the soft classification label and the reconstruction loss of the autoencoder are introduced into the clustering loss function as a regularization term to jointly refine the clustering. Compared with the experimental results of DEC method on datasets CIFAR10, CIFAR100 and STL10, ACC increaseds by 48.1, 23.1 and 41.8 percentage points, NMI increaseds by 41.0, 25.2 and 39.0 percentage points, and ARI increaseds by 45.4, 16.4 and 41.8 percentage points, respectively.

Key words: deep clustering, contrastive learning, autoencoder, vectorized feature space, embedding clustering

郑洋, 吴永明, 徐岸. 基于对比学习的矢量化特征空间嵌入聚类[J]. 计算机工程与应用, 2024, 60(4): 211-219.

ZHENG Yang, WU Yongming, XU An. Vectorized Feature Space Embedded Clustering Based on Contrastive Learning[J]. Computer Engineering and Applications, 2024, 60(4): 211-219.

参考文献

[1] SAEEDI EMADI H, MAZINANI S M. A novel anomaly detection algorithm using DBSCAN and SVM in wireless sensor networks[J]. Wireless Personal Communications, 2018, 98(2): 2025-2035.
[2] WANG M, DENG W. Deep face recognition with clustering based domain adaptation[J]. Neurocomputing, 2020, 393: 1-14.
[3] HOU H, DING S, XU X. A deep clustering by multi-level feature fusion[J]. International Journal of Machine Learning and Cybernetics, 2022, 13: 2813-2823.
[4] MACQUEEN J. Some methods for classification and analysis of multivariate observations[C]//Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability, 1967: 281-297.
[5] 彭长根, 高婷, 刘惠篮, 等. 面向机器学习模型的基于PCA的成员推理攻击[J]. 通信学报, 2022, 43(1): 149-160.
PENG C G, GAO T, LIU H L, et al. PCA-based membership inference attack for machine learning models[J]. Journal on Communications, 2022, 43(1): 149-160.
[6] SUN L, MA C, CHEN Y, et al. Low rank component induced spatial-spectral kernel method for hyperspectral image classification[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2019, 30(10): 3829-3842.
[7] TURCHETTI C, FALASCHETTI L. A manifold learning approach to dimensionality reduction for modeling data[J]. Information Sciences, 2019, 491: 16-29.
[8] ZHOU S, XU H, ZHENG Z, et al. A comprehensive survey on deep clustering: taxonomy, challenges, and future directions[J]. arXiv:2206.07579, 2022.
[9] 陶文彬, 钱育蓉, 张伊扬, 等. 基于自编码器的深度聚类算法综述[J]. 计算机工程与应用, 2022, 58(18): 16-25.
TAO W B, QIAN Y R, ZHANG Y Y, et al. Survey of deep clustering algorithm based on autoencoder[J]. Computer Engineering and Applications, 2022, 58(18): 16-25.
[10] BENGIO Y, LAMBLIN P, POPOVICI D, et al. Greedy layer-wise training of deep networks[C]//Proceedings of the 19th International Conference on Neural Information Processing Systems, 2006: 153-160.
[11] KINGMA D P, WELLING M. Auto-encoding variational Bayes[J]. arXiv:1312.6114, 2013.
[12] XIE J, GIRSHICK R, FARHADI A. Unsupervised deep embedding for clustering analysis[C]//Proceedings of the 33rd International Conference on Machine Learning, 2016: 478-487.
[13] YANG J, PARIKH D, BATRA D. Joint unsupervised learning of deep representations and image clusters[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 5147-5156.
[14] JI X, HENRIQUES J F, VEDALDI A. Invariant information clustering for unsupervised image classification and segmentation[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, 2019: 9865-9874.
[15] HUANG J, GONG S, ZHU X. Deep semantic clustering by partition confidence maximisation[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 8849-8858.
[16] GUO X, LIU X, ZHU E, et al. Deep clustering with convolutional autoencoders[C]//Proceedings of the 24th International Conference on Neural Information Processing. Cham: Springer, 2017: 373-382.
[17] GUO X, ZHU E, LIU X, et al. Deep embedded clustering with data augmentation[C]//Proceedings of the 10th Asian Conference on Machine Learning, 2018: 550-565.
[18] CHEN P, LIU S, JIA J. Jigsaw clustering for unsupervised visual representation learning[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 11526-11535.
[19] GIDARIS S, SINGH P, KOMODAKIS N. Unsupervised representation learning by predicting image rotations[J]. arXiv:1803.07728, 2018.
[20] MISRA I, MAATEN L. Self-supervised learning of pretext-invariant representations[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 6707-6717.
[21] CHEN T, KORNBLITH S, NOROUZI M, et al. A simple framework for contrastive learning of visual representations[C]//Proceedings of the 37th International Conference on Machine Learning, 2020: 1597-1607.
[22] ZELNIK-MANOR L, PERONA P. Self-tuning spectral clustering[C]//Advances in Neural Information Processing Systems, 2004: 1601-1608.
[23] CAI D, HE X, WANG X, et al. Locality preserving nonnegative matrix factorization[C]//Proceedings of the 21st International Joint Conference on Artificial Intelligence, 2009: 1010-1015.
[24] VINCENT P, LAROCHELLE H, LAJOIE I, et al. Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion[J]. Journal of Machine Learning Research, 2010, 11: 3371-3408.
[25] SCHERER D, MüLLER A, BEHNKE S. Evaluation of pooling operations in convolutional architectures for object recognition[C]//Proceedings of the 20th International Conference on Artificial Neural Networks. Berlin, Heidelberg: Springer, 2010: 92-101.
[26] RADFORD A, METZ L, CHINTALA S. Unsupervised representation learning with deep convolutional generative adversarial networks[J]. arXiv:1511.06434, 2015.
[27] PATACCHIOLA M, STORKEY A J. Self-supervised relational reasoning for representation learning[C]//Advances in Neural Information Processing Systems 33, 2020: 4003-4014.
[28] GOWDA K C, KRISHNA G. Agglomerative clustering using the concept of mutual nearest neighbourhood[J]. Pattern Recognition, 1978, 10(2): 105-112.
[29] CAI Y, ZHANG Z, CAI Z, et al. Graph convolutional subspace clustering: a robust subspace clustering framework for hyperspectral image[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 59(5): 4191-4202.
[30] CHANG J, WANG L, MENG G, et al. Deep adaptive image clustering[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, 2017: 5879-5887.
[31] WU J, LONG K, WANG F, et al. Deep comprehensive correlation mining for image clustering[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, 2019: 8150-8159.
[32] YU Y, CHAN K H R, YOU C, et al. Learning diverse and discriminative representations via the principle of maximal coding rate reduction[C]//Advances in Neural Information Processing Systems 33, 2020: 9422-9434.