Semantic Segmentation Method for Remote Sensing Images Based on Improved Swin Transformer

doi:10.3778/j.issn.1002-8331.2310-0302

Abstract

Abstract: Extracting accurate feature information in high-resolution remote sensing images plays an important role in urban planning as well as land resource utilization. However, remote sensing images are characterized by large scale differences between target objects and complex backgrounds, which easily lead to inaccurate extraction results, especially the low extraction accuracy for small-scale features. In order to solve these problems, this paper proposes a novel dual-coding structure to fully acquire global semantic information as well as spatial detail information, and to fuse feature information at different scales in stages to enhance the feature representation capability. The feature enhancement module (FEM) is constructed to reduce the loss of detail information in downsampling and focus on more small-scale features. In order to better refine the feature information, channel attention and kernel attention are fused and then up-sampled, which is able to integrate the local features with the corresponding global spatial dependencies and enhance the segmentation accuracy of the target object. The mIoU on Potsdam dataset and Vaihingen dataset are 86.1% and 82.4%, respectively. Comparative analysis with popular semantic segmentation models shows that the method in this paper can effectively solve the problem of inaccurate segmentation of small- and medium-scale objects in remote sensing images, and it is suitable for dealing with the task of semantic segmentation of remote sensing images.

Key words: semantic segmentation, dual coding structure, feature enhancement, fusion attention mechanism, small scale features

摘要： 在高分辨率的遥感图像中提取出准确的地物信息对城市规划以及土地资源利用有重要作用。然而，遥感图像具有目标物体之间尺度差异大，背景复杂等特点，易导致提取结果不准确，特别是对小尺度地物的提取精度较低。为了解决这些问题，提出一种新型双编码结构，充分获取全局语义信息以及空间细节信息，分阶段融合不同尺度的特征信息，增强特征表示能力。构造了特征加强模块（FEM），以减少下采样中细节信息的丢失，关注更多小尺度特征。为了更好地细化特征信息，融合了通道注意力和内核注意力后进行上采样，能够将局部特征与对应的全局空间依赖关系整合，提升目标物体的分割精度。在Potsdam数据集和Vaihingen数据集上的mIoU分别为86.1%和82.4%，与流行的语义分割模型进行对比分析，结果表明，该方法能够有效解决遥感图像中小尺度物体分割不准确的问题，适合处理遥感图像语义分割任务。

关键词: 语义分割, 双编码结构, 特征加强, 融合注意力机制, 小尺度地物

WANG Yizhong, HU Yaqi, WU Xiaosuo, YAN Haowen, WANG Xiaocheng. Semantic Segmentation Method for Remote Sensing Images Based on Improved Swin Transformer[J]. Computer Engineering and Applications, 2024, 60(11): 194-203.

王一中, 胡亚琦, 吴小所, 闫浩文, 王小成. 基于改进Swin Transformer的遥感图像语义分割方法[J]. 计算机工程与应用, 2024, 60(11): 194-203.

References

[1] LIN A, CHEN B, XU J, et al. DS-TransUNet: dual swin Transformer U-Net for medical image segmentation[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1-15.
[2] ZHANG L, MA J, LV X, et al. Hierarchical weakly supervised learning for residential area semantic segmentation in remote sensing images[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 17(1): 117-121.
[3] ZHAO J, ZHOU Y, SHI B, et al. Multi-stage fusion and multi-source attention network for multi-modal remote sensing image segmentation[J]. ACM Transactions on Intelligent Systems and Technology (TIST), 2021, 12(6): 1-20.
[4] SHEIKH R, MILIOTO A, LOTTES P, et al. Gradient and log-based active learning for semantic segmentation of crop and weed for agricultural robots[C]//Proceedings of the 2020 IEEE International Conference on Robotics and Automation, 2020: 1350-1356.
[5] ZHANG X, XIAO Z, LI D, et al. Semantic segmentation of remote sensing images using multiscale decoding network[J]. IEEE Geoscience and Remote Sensing Letters, 2019, 16(9): 1492-1496.
[6] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 3431-3440.
[7] YI Y N, ZHANG Z J, ZHANG W C, et al. Semantic segmentation of urban buildings from VHR remote sensing imagery using a deep convolutional neural network[J]. Remote Sensing, 2019, 10(15): 1774-1792.
[8] MINAEE S, BOYKOV Y Y, PORIKLI F, et al. Image segmentation using deep learning: a survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(7): 3523-3542.
[9] CHEN L C, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation[J]. arXiv:1706.05587, 2017.
[10] 田雪伟, 汪佳丽, 陈明, 等. 改进 SegFormer 网络的遥感图像语义分割方法[J]. 计算机工程与应用, 2023, 59(8): 217-226.
TIAN X W, WANG J L, CHEN M, et al. Semantic segmentation of remote sensing image based on improved SegFormer network[J]. Computer Engineering and Applications, 2023, 59(8): 217-226.
[11] RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of the Medical Image Computing and Computer-Assisted Intervention (MICCAI 2015), 2015: 234-241.
[12] RASTOGI K, BODANI P, SHARMA S A. Automatic building footprint extraction from very high-resolution imagery using deep learning techniques[J]. Geocarto International, 2022, 37(5): 1501-1513.
[13] 项剑文, 陈泯融, 杨百冰. 结合Swin及多尺度特征融合的细粒度图像分类[J]. 计算机工程与应用, 2023, 59(20): 147-157.
XIANG J W, CHEN M R, YANG B B. Fine-grained image classification combined with Swin and multi-scale feature fusion[J]. Computer Engineering and Applications, 2023, 59(20): 147-157.
[14] FU J, LIU J, TIAN H, et al. Dual attention network for scene segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 3146-3154.
[15] LI R, ZHENG S, ZHANG C, et al. Multiattention network for semantic segmentation of fine-resolution remote sensing images[J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 60: 1-13.
[16] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognition at scale[J]. arXiv:2010.11929, 2020.
[17] LIN H, CHENG X, WU X, et al. CAT: cross attention in vision transformer[C]//Proceedings of the 2022 IEEE International Conference on Multimedia and Expo, 2022: 1-6.
[18] HE X, ZHOU Y, ZHAO J, et al. Swin transformer embedding UNet for remote sensing image semantic segmentation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 1-15.
[19] GAO L, LIU H, YANG M, et al. STransFuse: fusing swin transformer and convolutional neural network for remote sensing image semantic segmentation[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14: 10990-11003.
[20] WANG L, LI R, DUAN C, et al. A novel transformer based semantic segmentation scheme for fine-resolution remote sensing images[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 1-5.
[21] ALMARZOUQI H, SAOUD L S. Semantic labeling of high resolution images using EfficientUNets and Transformers[J]. IEEE Transactions on Geoscience and Remote Sensing, 2023, 61: 1-13.
[22] LIU Z, LIN Y, CAO Y, et al. Swin Transformer: hierarchical visiontransformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 10012-10022.
[23] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
[24] DAI Y, GIESEKE F, OEHMCKE S, et al. Attentional feature fusion[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021: 3560-3569.
[25] YUAN Y, HUANG L, GUO J, et al. OCNet: object context for semantic segmentation[J]. International Journal of Computer Vision, 2021, 129(8): 2375-2398.
[26] WANG L, LI R, WANG D, et al. Transformer meets convolution: a bilateral awareness network for semantic segmentation of very fine resolution urban scene images[J]. Remote Sensing, 2021, 13(16): 3065.
[27] CHEN J, ZHANG D, WU Y, et al. A context feature enhancement network for building extraction from high-resolution remote sensing imagery[J]. Remote Sensing, 2022, 14(9): 2276.