三维卷积与Transformer支持下联合空谱特征的高光谱影像分类

doi:10.3778/j.issn.1002-8331.2308-0401

摘要/Abstract

摘要： 由于CNN对局部特征提取能力强，目前仍是高光谱影像处理和分析中的主流深度模型，但是CNN感受野有限，无法建立长距离依赖关系，学习全局语义信息受限。Transformer的自注意力机制可以对输入序列中的每个位置进行注意力计算，从而能有效获取全局上下文信息。如何实现CNN和Transformer的技术耦合并充分利用空间信息和光谱信息进行高光谱遥感影像分类是一个重要的待研问题。鉴于此，提出一种新的基于三维卷积和Transformer的高光谱遥感影像分类方法，尝试联合空谱特征实现解译能力的提升。使用主成分分析方法对高光谱遥感影像沿垂直方向降维；用非负矩阵分解算法对降维后遥感影像沿水平方向进行空间特征提取，将两种工具处理后遥感影像进行拼接，以充分保留信息；再用三维卷积核对拼接后遥感影像进行空间特征和光谱特征的综合提取；用Transformer的注意力机制对提取空间信息和光谱信息的遥感影像序列建立长距离依赖关系并使用多层感知机完成分类任务。实验表明，所提方法在WHU-Hi龙口、汉川、洪湖以及雄安新区马蹄湾村数据集上均表现出比对比方法更优异的分类性能，表明该方法具有一定的泛化性和稳健性。

关键词: 非负矩阵分解, 特征融合, 三维卷积, 空谱联合, Transformer, 高光谱遥感影像分类

Abstract: Due to its strong ability to extract local features, CNN is still the mainstream depth model in hyperspectral image processing and analysis. However, CNN has limited receptive field, cannot establish long-distance dependence, and is limited in learning global semantic information. Transformer’s self-attention mechanism can perform attention calculations at each position in the input sequence to effectively capture global context information. How to realize the technical coupling of CNN and Transformer and make full use of spatial and spectral information for hyperspectral remote sensing image classification is an important problem to be studied. In view of this, a new hyperspectral remote sensing image classification method based on 3D convolution and Transformer is proposed in this paper, which attempts to improve the interpretation ability by combining spatial spectral features. Firstly, the principal component analysis method is used to reduce the dimensionality of hyperspectral remote sensing images in the vertical direction. Then the spatial features of the remote sensing images after dimensionality reduction are extracted along the horizontal direction by non-negative matrix decomposition algorithm, and then the remote sensing images processed by the two tools are combined to fully retain the information. Then the three-dimensional convolution check is used to extract the spatial and spectral features of the remote sensing images. Finally, the attention mechanism of Transformer is used to establish long-distance dependencies on remote sensing image sequences that extract spatial and spectral information, and a multi-layer perceptron is used to complete classification tasks. Experiment shows that the proposed method exhibits better classification performance than the comparison method on the WHU-Hi Longkou, Hanchuan, Honghu, and Matiwan Village datasets in Xiong’an New Area, indicating that the proposed method has a certain degree of generalization and robustness.

Key words: nonnegative matrix factorization, feature fusion, 3D convolution, spatial spectral combination, Transformer, hyperspectral remote sensing image classification

何光, 吴田军. 三维卷积与Transformer支持下联合空谱特征的高光谱影像分类[J]. 计算机工程与应用, 2025, 61(2): 259-272.

HE Guang, WU Tianjun. Hyperspectral Image Classification Employing Spatial-Spectral Feature Supported by 3D Convolution and Transformer[J]. Computer Engineering and Applications, 2025, 61(2): 259-272.

参考文献

[1] LONGBOTHAM N, CHAAPEL C, BLEILER L, et al. Very high resolution multiangle urban classification analysis[J]. IEEE Transactions on Geoscience and Remote Sensing, 2011, 50(4): 1155-1170.
[2] TAYYEBI A, PIJANOWSKI B C, TAYYEBI A H. An urban growth boundary model using neural networks, GIS and radial parameterization: an application to Tehran, Iran[J]. Landscape and Urban Planning, 2011, 100(1/2): 35-44.
[3] HUANG X, WEN D W, LI J Y, et al. Multi-level monitoring of subtle urban changes for the megacities of China using high-resolution multi-view satellite imagery[J]. Remote Sensing of Environment, 2017, 196: 56-75.
[4] WANG Y B, ZHANG L Q, TONG X H, et al. A three-layered graph-based learning approach for remote sensing image retrieval[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016, 54(10): 6020-6034.
[5] MISHRA N B, CREWS K A. Mapping vegetation morphology types in a dry savanna ecosystem: integrating hierarchical object-based image analysis with random forest[J]. International Journal of Remote Sensing, 2014, 35(3): 1175-1198.
[6] MELGANI F, BRUZZONE L. Classification of hyperspectral remote sensing images with support vector machines[J]. IEEE Transactions on Geoscience and Remote Sensing, 2004, 42(8): 1778-1790.
[7] MA L, CRAWFORD M M, TIAN J. Local manifold learning-based k-nearest-neighbor for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2010, 48(11): 4099-4109.
[8] DALPONTE M, ?RKA O H, GOBAKKEN T, et al. Tree species classification in boreal forests with hyperspectral data[J]. IEEE Transactions on Geoscience and Remote Sensing, 2013, 51(5): 2632-2645.
[9] GHAMISI P, PLAZA J, CHEN Y, et al. Advanced spectral classifiers for hyperspectral images: a review[J]. IEEE Geoscience and Remote Sensing Magazine, 2017, 5(1): 8-32.
[10] 李玥, 罗滔. 深度学习理论的高光谱图像分类方法[J]. 激光杂志, 2020, 41(9): 221-224.
LI Y, LUO T. Hyperspectral image classification based on deep learning theory[J]. Laser Journal, 2020, 41(9): 221-224.
[11] 赵红伟, 陈仲新, 刘佳. 深度学习方法在作物遥感分类中的应用和挑战[J]. 中国农业资源与区划, 2020, 41(2): 35-49.
ZHAO H W, CHEN Z X, LIU J. Deep learning for crop classification of remote sensing data: applications and challenges[J]. Journal of China Agricultural Resources and Regional Planning, 2020, 41(2): 35-49.
[12] TONG L, ZHANG J, ZHANG Y. Classification of hyperspectral image based on deep belief networks[C]//Proceedings of the IEEE International Conference on Image Processing, 2014: 5132-5136.
[13] XU Q, XIAO Y, WANG D, et al. CSA-MSO3DCNN: multiscale octave 3D CNN with channel and spatil attention for hyperspectral image classification[J]. Remote Sensing, 2020, 12(1): 188.
[14] GAO K, GUO W, YU X, et al. Deep induction network for small samples classification of hyperspectral images[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13: 3462-3477.
[15] HU W, HUANG Y Y, WEI L, et al. Deep convolutional neural networks for hyperspectral image classification[J]. Journal of Sensors, 2015, 2015: 1-12.
[16] ZHAO W Z, DU S H. Learning multiscale and deep representations for classifying remotely sensed imagery[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2016, 113: 155-165.
[17] YUE J, ZHAO W, MAO S, et al. Spectral-spatial classification of hyperspectral images using deep convolutional neural networks[J]. Remote Sensing Letters, 2015, 6(4/5/6): 468-477.
[18] CHAMISI P, CHEN Y, XIAO X Z. A self-improving convolution neural network for the classification of hyperspectral data[J]. IEEE Geoscience and Remote Sensing Letters, 2016, 13(10): 1537-1541.
[19] LI Y, ZHANG H K, SHEN Q. Spectral-spatial classification of hyperspectral imagery with 3D convolutional neural network [J]. Remote Sensing, 2017, 9(1): 67.
[20] ROY S K, KRISHNA G, et al. HybridSN: exploring 3-D-2-D CNN feature hierarchy for hyperspectral image classification[J]. IEEE Geoscience and Remote Sensing Letters, 2020, 17(2): 277-281.
[21] HE X, CHEN Y. Transferring CNN ensemble for hyperspectral image classification[J]. IEEE Geoscience and Remote Sensing Letters, 2021, 18(5): 876-880.
[22] MOU L, GHAMISI P, ZHU X X. Unsupervised spectral-spatial feature learning via deep residual Conv-Deconv network for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(1): 391-406.
[23] MULLER G, RIOS M, SENNRICH A, et al. Why self-attention? a targeted evaluation of neural machine translation architectures[J]. arXiv:1808.08946, 2018.
[24] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognition at scale[J]. arXiv:2010.11929, 2020.
[25] 李清格, 杨小冈, 卢瑞涛, 等. 计算机视觉中的Transformer发展综述[J]. 小型微型计算机系统, 2023, 44(4): 850-861.
LI Q G, YANG X G, LU R T, et al. Transformer in computer vision: a survey[J]. Journal of Chinese Mini-Micro Computer Systems, 2023, 44(4): 850-861.
[26] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 6000-6010.
[27] QING Y H, LIU W Y, FENG L Y, et al. Improved transformer net for hyperspectral image classification[J]. Remote Sensing, 2021, 13(11): 2216.
[28] HONG D F, HAN Z, YAO J, et al. SpectralFormer: rethinking hyperspectral image classi?cation with transformers[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 1-15.
[29] MEI S H, SONG C, MA M Y, et al. Hyperspectral image classification using group-aware hierarchical transformer[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-14.
[30] 张艺超, 郑向涛, 卢孝强. 基于层级Transformer的高光谱图像分类方法[J]. 测绘学报, 2023, 52(7): 1139-1147.
ZHANG Y C, ZHENG X T, LU X Q. Hyperspectral image classification method based on hierarchical trans-former network[J]. Journal of Geodesy and Geoinformation Science, 2023, 52(7): 1139-1147.
[31] 叶珍, 白遴, 何明一. 高光谱图像空谱特征提取综述[J]. 中国图象图形学报, 2021, 26(8): 1737-1763.
YE Z, BAI L, HE M Y. Review of spatial-spectral feature extraction for hyperspectral images[J]. Journal of Image and Graphics, 2021, 26(8): 1737-1763.
[32] LEE D D, SEUNG H S. Algorithms for non-negative matrix factorization[C]//Advances in Neural Information Processing Systems, 2001: 556-562.
[33] ZHONG Y F, HU Y, LUO C, et al. WHU-Hi: UAV-borne hyperspectral with high spatial resolution (H2) benchmark datasets and classifier for precise crop identification based on deep convolutional neural network with CRF[J]. Remote Sensing, 2020, 250: 112012.
[34] ZHONG Y F, WANG X Y, XU Y, et al. Mini-UAV-borne hyperspectral remote sensing: from observation and processing to applications[J]. IEEE Geoscience and Remote Sensing Magazine, 2018, 6(4): 46-62.
[35] 岑奕, 张立福, 张霞, 等. 雄安新区马蹄湾村航空高光谱遥感影像分类数据集[J]. 遥感学报, 2020, 24(11): 1299-1306.
CEN Y, ZHANG L F, ZHANG X, et al. Aerial hyperspectral remote sensing classification dataset of Xiongan new area (Matiwan Village)[J]. Journal of Remote Sensing, 2020, 24(11): 1299-1306.