Hyperspectral Image Classification Based on Double Branch Multidimensional Attention Feature Fusion

doi:10.3778/j.issn.1002-8331.2211-0139

Abstract

Abstract: To improve the classification performance of small sample classes of hyperspectral images and to enhance the robustness of the model feature representation, a neural network classification model with two-branch multidimensional attentional feature fusion (DBMD) is proposed. DBMD uses two branches for spectral feature extraction and hybrid feature extraction respectively. The spectral branch extracts features step-by-step through densely connected dilated convolution, and then fuses low, medium and high level semantic information as the feature output. The hybrid branch uses a 3D-2D network architecture and extracts spatial scale features through improved Inception blocks. In addition, the attention mechanism is applied to spectral, spatial and spatial-spectral feature extraction respectively for feature refinement and to enhance the feature response in important regions. Finally, the refined features of different dimensions are jointly input to the classifier for classification. Experiments using 5% and 1% samples on the Indian Pines and Salinas Valley datasets achieve an overall accuracy of 98.40% and 99.78% respectively, and the proposed model performs better in terms of accuracy and stability compared to the other six network architectures.

Key words: hybrid feature extraction, attention mechanism, multidimensional feature fusion, image classification

摘要： 为改善高光谱图像小样本类别的分类性能，提高模型特征表达的稳健性，提出了双分支多维注意力特征融合的神经网络分类模型（DBMD）。DBMD采用两个分支分别进行光谱特征提取和混合特征提取。光谱分支通过密集连接的扩张卷积逐级提取特征，然后融合低、中、高级语义信息作为特征输出。混合分支采用3D-2D网络架构，并通过改进的Inception块提取空间尺度特征。此外，注意力机制分别应用于光谱、空间和空谱特征，进行特征细化，增强重要区域的特征响应。最后，将不同维度的细化特征联合输入至分类器进行分类。在Indian Pines和Salinas Valley数据集上利用5%和1%的样本进行实验，分别取得了98.40%和99.78%的总体精度，与其他六种网络架构相比，该模型在准确性和稳定性上都具有更优的表现。

关键词: 混合特征提取, 注意力机制, 多维特征融合, 图像分类

MA Yamei, WANG Shuangting, DU Weibing. Hyperspectral Image Classification Based on Double Branch Multidimensional Attention Feature Fusion[J]. Computer Engineering and Applications, 2024, 60(7): 192-203.

马亚美, 王双亭, 都伟冰. 双分支多维注意特征融合的高光谱图像分类[J]. 计算机工程与应用, 2024, 60(7): 192-203.

References

[1] ACOSTA I C C, KHODADADZADEH M, TUSA L, et al. A machine learning framework for drill-core mineral mapping using hyperspectral and high-resolution mineralogical data fusion[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12(12): 4829-4842.
[2] LU B, DAO P, LIU J G, et al. Recent advances of hyperspectral imaging technology and applications in agriculture[J]. Remote Sensing, 2020, 12(16): 2659.
[3] 谢东津, 吕呈龙, 祖梅, 等. 绿色植被可见-近红外反射光谱模拟材料研究进展[J]. 光谱学与光谱分析, 2021, 41(4): 1032-1038.
XIE D J, LV C L, ZU M, et al. Research progress of bionic materials simulating vegetation visible-near infrared reflectance spectra[J]. Spectroscopy and Spectral Analysis, 2021, 41(4): 1032-1038.
[4] KARALAS K, TSAGKATAKIS G, ZERVAKIS M, et al. Land classification using remotely sensed data: going multilabel[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016, 54(6): 3548-3563.
[5] BIOUCAS-DIAS J M, PLAZA A, CAMPS-VALLS G, et al. Hyperspectral remote sensing data analysis and future challenges[J]. IEEE Geoscience and Remote Sensing Magazine, 2013, 1(2): 6-36.
[6] 叶珍, 白璘, 何明一. 高光谱图像空谱特征提取综述[J]. 中国图象图形学报, 2021, 26(8): 1737-1763.
YE Z, BAI L, HE M Y. Review of spatial-spectral feature extraction for hyperspectral image[J]. Journal of Image and Graphics, 2021, 26(8): 1737-1763.
[7] 高伟, 李维良, 林妍. 面向高光谱影像分类的高性能计算及存储优化[J]. 计算机工程与应用. 2015, 51(16): 171-177.
GAO W, LI W L, LIN Y. High performance computing and its storage optimization strategies oriented to hyperspectral image classification[J]. Computer Engineering and Applications, 2015, 51(16): 171-177.
[8] MELGANI F, BRUZZONE L. Classification of hyperspectral remote sensing images with support vector machines[J]. IEEE Transactions on Geoscience and Remote Sensing, 2004, 42(8): 1778-1790.
[9] LI J, BIOUCAS-DIAS J M, PLAZA A. Spectral-spatial hyperspectral image segmentation using subspace multinomial logistic regression and Markov random fields[J]. IEEE Transactions on Geoscience and Remote Sensing, 2012, 50(3): 809-823.
[10] ZHAO J, JIANG Q. Probabilistic PCA for t distributions[J]. Neurocomputing, 2006, 69(16): 2217-2226.
[11] KUN Z, LAI-WAN C. Dimension reduction as a deflation method in ICA[J]. IEEE Signal Processing Letters, 2006, 13(1): 45-48.
[12] JIANG H, DONG Y. Dimension reduction based on a penalized kernel support vector machine model[J]. Knowledge-Based Systems, 2017, 138: 79-90.
[13] LAOHAKIAT S, PHIMOLTARES S, LURSINSAP C. A clustering algorithm for stream data with LDA-based unsupervised localized dimension reduction[J]. Information Sciences, 2017, 381: 104-123.
[14] 吴昊. 综合纹理特征的高光谱遥感图像分类方法[J]. 计算机工程与设计, 2012, 33(5): 1993-1996.
WU H. Classification methodology combined with texture feature for hyperspectral remote sensing image[J]. Computer Engineering and Design, 2012, 33(5): 1993-1996.
[15] MA L, MA A D, JU C, et al. Graph-based semi-supervised learning for spectral-spatial hyperspectral image classification[J]. Pattern Recognition Letters, 2016, 83: 133-142.
[16] 李垒, 任越美. 基于随机森林的高光谱遥感图像分类[J]. 计算机工程与应用, 2016, 52(24): 189-193.
LI L, REN Y M. Classification of hyperspectral data based on random forest[J]. Computer Engineering and Applications, 2016, 52(24): 189-193.
[17] ZHANG X R, SONG Q, GAO Z Y, et al. Spectral-spatial feature learning using cluster-based group sparse coding for hyperspectral image classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2016, 9(9): 4142-4159.
[18] 安凤平, 李晓薇, 曹翔. 权重初始化-滑动窗口CNN的医学图像分类[J]. 计算机科学与探索, 2022, 16(8): 1885-1897.
AN F P, LI X W, CAO X. Medical image classification algorithm based on weight initialization-sliding window CNN[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(8): 1885-1897.
[19] 朱炳宇, 刘朕, 张景祥. 融合Grad-CAM和卷积神经网络的COVID-19检测算法[J]. 计算机科学与探索, 2022, 16(9): 2108-2120.
ZHU B Y, LIU Z, ZHANG J X. COVID-19 detection algorithm combining Grad-CAM and convolutional neural network[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(9): 2108-2120.
[20] AL-GARADI M A, YANG Y C, SARKER A. The role of natural language processing during the COVID-19 pandemic: health applications, opportunities, and challenges[J]. Healthcare, 2022, 10(11): 2270.
[21] 焦磊, 云静, 刘利民, 等. 封闭域深度学习事件抽取方法研究综述[J]. 计算机科学与探索, 2023, 17(3): 533-548.
JIAO L, YUN J, LIU L M, et al. Overview of closed-domain deep learning event extraction methods[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(3): 533-548.
[22] CHEN Y S, LIN Z H, ZHAO X, et al. Deep learning-based classification of hyperspectral data[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2014, 7(6): 2094-2107.
[23] LI T, ZHANG J P, ZHANG Y. Classification of hyperspectral image based on deep belief networks[C]//IEEE International Conference on Image Processing (ICIP), 2014: 5132-5136.
[24] HU W, HUANG Y Y, LI W, et al. Deep convolutional neural networks for hyperspectral image classification[J]. Journal of Sensors, 2015: 1-12.
[25] MOU L C, GHAMISI P, ZHU X X. Deep recurrent neural networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(7): 3639-3655.
[26] HE Z, LIU H, WANG Y W, et al. Generative adversarial networks-based semi-supervised learning for hyperspectral image classification[J]. Remote Sensing, 2017, 9(10): 1042.
[27] CHEN Y S, JIANG H L, LI C Y, et al. Deep feature extraction and classification of hyperspectral images based on convolutional neural networks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2016, 54(10): 6232-6251.
[28] LEE H, KWON H. Going deeper with contextual CNN for hyperspectral image classification[J]. IEEE Transactions on Image Processing, 2017, 26(10): 4843-4855.
[29] ZHONG Z L, LI J, LUO Z M, et al. Spectral-spatial residual network for hyperspectral image classification: a 3-D deep learning framework[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(2): 847-858.
[30] ROY S K, KRISHNA G, DUBEY S R, et al. HybridSN: exploring 3-D-2-D CNN feature hierarchy for hyperspectral image classification[J]. IEEE Geoscience and Remote Sensing Letters, 2020, 17(2): 277-281.
[31] LI Z K, WANG T N, LI W, et al. Deep multilayer fusion dense network for hyperspectral image classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13: 1258-1270.
[32] 魏祥坡, 余旭初, 管凌霄. 利用残差通道注意力网络的高光谱图像分类[J]. 测绘科学技术学报, 2019, 36(2): 161-166.
WEI X P, YU X C, GUAN L X. Hyperspectral image classification using residual channel attention network[J]. Journal of Geomatics Science and Technology, 2019, 36(2): 161-166.
[33] MA W P, YANG Q F, WU Y, et al. Double-branch multi-attention mechanism network for hyperspectral image classification[J]. Remote Sensing, 2019, 11(11): 1307.
[34] LI W, WU G D, ZHANG F, et al. Hyperspectral image classification using deep pixel-pair features[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(2): 844-853.
[35] LI W, CHEN C, ZHANG M M, et al. Data augmentation for hyperspectral image classification with deep CNN[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 16(4): 593-597.
[36] ZHU L, CHEN Y S, GHAMISI P, et al. Generative adversarial networks for hyperspectral image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(9): 5046-5063.
[37] FANG B, LI Y, ZHANG H K, et al. Collaborative learning of lightweight convolutional neural network and deep clustering for hyperspectral image semi-supervised classification with limited training samples[J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 161: 164-178.
[38] WANG Q Y, CHEN M, ZHANG J P, et al. Improved active deep learning for semi-supervised classification of hyperspectral image[J]. Remote Sensing, 2022, 14(1): 171.
[39] SUN J G, WANG W L, WEI X, et al. Deep clustering with intraclass distance constraint for hyperspectral images[J]. IEEE Transactions on Geoscience and Remote sensing, 2021, 59(5): 4135-4149.
[40] 石延新, 何进荣, 李照奎, 等. 3D卷积自编码器高光谱图像分类模型[J]. 中国图象图形学报, 2021, 26(8): 2021-2036.
SHI Y X, HE J R, LI Z K, et al. Hyperspectral image classification model based on 3D convolutional auto-encoder[J]. Journal of Image and Graphics, 2021, 26(8): 2021-2036.
[41] FENG F, ZHANG Y S, ZHANG J, et al. Small sample hyperspectral image classification based on cascade fusion of mixed spatial-spectral features and second-order pooling[J]. Remote Sensing, 2022, 14(3): 505.
[42] LIU J X, ZHANG K F, WU S Q, et al. An investigation of a multidimensional CNN combined with an attention mechanism model to resolve small-sample problems in hyperspectral image classification[J]. Remote Sensing, 2022, 14(3): 785.
[43] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//IEEE Conference on Computer Vision & Pattern Recognition, Las Vegas, NV, USA, 2016: 770-778.
[44] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017: 2261-2269.
[45] YU F, KOLTUN V. Multi-scale context aggregation by dilated convolutions[C]//Proceedings of International Conference on Learning Representation (ICLR), San Diego, CA, USA, 2016: 7-9.
[46] SZEGEDY C, LIU W, JIA Y Q, et al. Going deeper with convolutions[C]//IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, 2015: 1-9.
[47] HOWARD A, ZHU M L, CHEN B, et al. MobileNets: efficient convolutional neural networks for mobile vision applications[J]. arXiv:1704.04861, 2017.
[48] DONG H W, ZHANG L M, ZOU B. Band attention convolutional networks for hyperspectral image classification[J]. arXiv:1906.04379, 2019.
[49] SONG W W, LI S T, FANG L Y, et al. Hyperspectral image classification with deep feature fusion network[J]. IEEE Transactions on Geoscience and Remote Sensing, 2018, 56(6): 3173-3184.
[50] WANG W J, DOU S G, JIANG Z M, et al. A fast dense spectral-spatial convolution network framework for hyperspectral images classification[J]. Remote Sensing, 2018, 10(7): 1068.
[51] LI Z K, ZHAO X D, XU Y M, et al. Hyperspectral image classification with multiattention fusion network[J]. IEEE Geoscience and Remote Sensing Letters, 2020: 1-5.