自适应卷积和轻量化Transformer的遥感图像超分辨网络

doi:10.3778/j.issn.1002-8331.2312-0185

摘要/Abstract

摘要： 随着深度学习的发展，基于卷积的图像超分辨在遥感领域取得了广泛应用，而卷积往往对局部特征更加敏感，难以提取遥感图像全局特征。Transformer因其自注意力机制能够对全局上下文进行特征提取而更具优势，但其高空间时间复杂度也给遥感图像处理有限的计算资源带来了不可忽视的负担。针对以上问题，提出一种自适应卷积与轻量化Transformer相结合的遥感图像超分辨网络。提出基于卷积的自适应局部特征提取，通过自适应权重动态融合高低频图像特征中不同感受野范围的细粒度局部特征。提出一种轻量化Transformer结构，结合遥感图像超分辨的任务特点将全局自注意力机制分解为全局信息引导下的局部自注意力机制，在保持全局感知能力的同时有效降低Transformer计算复杂度。实验结果表明，所提方法能够更好地重建遥感图像的纹理细节，实现计算复杂度和超分辨重建质量之间的良好权衡。

关键词: 图像超分辨, 遥感图像, 自适应卷积, 轻量化Transformer

Abstract: With the development of deep learning, convolution-based image super-resolution has been widely applied in remote sensing. However, convolution is biased towards local features, making it difficult to extract global features. Transformer has the advantage of its self-attention mechanism for feature extraction of the global context. However, the high space and time complexity brings a non-negligible burden to the limited computing resources in remote sensing image processing. To solve the above issues, a remote sensing image super-resolution network that combines adaptive convolution and lightweight Transformer is proposed. An adaptive local feature extraction based on convolution is proposed to dynamically fuse fine-grained local features with different receptive field ranges in high and low frequency image features through adaptive weights. In addition, a lightweight Transformer structure is proposed, which decomposes the global self-attention mechanism into a local self-attention mechanism weighted by global information based on the task characteristics of remote sensing image super-resolution. The lightweight Transformer effectively reduces the computational complexity while maintaining the global perception capability. Experimental results show that the proposed method can better reconstruct the texture details of remote sensing images and achieve a good trade-off between computational complexity and super-resolution reconstruction quality.

Key words: image super-resolution, remote sensing image, adaptive convolution, lightweight Transformer

罗宇轩, 吴高昌, 高明. 自适应卷积和轻量化Transformer的遥感图像超分辨网络[J]. 计算机工程与应用, 2025, 61(9): 263-276.

LUO Yuxuan, WU Gaochang, GAO Ming. Remote Sensing Image Super-Resolution Network with Adaptive Convolution and Lightweight Transformer[J]. Computer Engineering and Applications, 2025, 61(9): 263-276.

参考文献

[1] OKTAY O, BAI W, LEE M, et al. Multi-input cardiac image super-resolution using convolutional neural networks [C]//Proceedings of the 19th International Conference on Medical Image Computing and Computer-Assisted Intervention(MICCAI 2016), Athens, Greece, October 17-21, 2016: 246-254.
[2] LUO Y, ZHOU L, WANG S, et al. Video satellite imagery super resolution via convolutional neural networks [J]. IEEE Geoscience and Remote Sensing Letters, 2017(12): 2398-2402.
[3] RASTI P, UIBOUPIN T, ESCALERA S, et al. Convolutional neural network super resolution for face recognition in surveillance monitoring[C]//Proceedings of the 9th International Conference on Articulated Motion and Deformable Objects (AMDO 2016), Palma de Mallorca, Spain, July 13-15, 2016: 175-184.
[4] 温剑, 邵剑飞, 刘杰,等. 多维注意力机制与选择性特征融合的图像超分辨率重建[J]. 光学精密工程, 2023, 31(17): 2584-2597.
WEN J, SHAO J F, LIU J, et al. Multidimensional attention mechanism and selective feature fusion for image super-resolution reconstruction[J]. Optics and Precision Engineering,2023, 31(17): 2584-2597.
[5] LI W, WEI W, ZHANG L. GSDet: object detection in aerial images based on scale reasoning[J]. IEEE Transactions on Image Processing, 2021, 30: 4599-4609.
[6] HAUT J M, PAOLETTI M E, FERNANDEZ-BELTRAN R, et al. Remote sensing single-image superresolution based on a deep compendium model[J]. IEEE Geoscience and Remote Sensing Letters, 2019 (9): 1432-1436.
[7] DONG X, SUN X, JIA X, et al. Remote sensing image super-resolution using novel dense-sampling networks [J]. IEEE Transactions on Geoscience and Remote Sensing, 2020 (2): 1618-1633.
[8] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems, 2017.
[9] CHEN H, WANG Y, GUO T, et al. Pre-trained image processing transformer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 12299-12310.
[10] LIANG J, CAO J, SUN G, et al. SwinIR: Image restoration using swin transformer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 1833-1844.
[11] LIU Z, LIN Y, CAO Y, et al. Swin Transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 10012-10022.
[12] DONG C, LOY C C, HE K, et al. Image super-resolution using deep convolutional networks [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015(2): 295-307.
[13] 杨才东, 李承阳, 李忠博, 等. 深度学习的图像超分辨率重建技术综述[J]. 计算机科学与探索, 2022, 16(9): 1990-2010.
YANG C D, LI C Y, LI Z B, et al. Review of image super-resolution reconstruction algorithms based on deep learning[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(9): 1990-2010.
[14] KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1646-1654.
[15] ZHANG Y, TIAN Y, KONG Y, et al. Residual dense network for image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 2472-2481.
[16] ZHANG Y L, LI K P, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]// Proceedings of the European Conference on Computer Vision (ECCV), 2018: 286-301.
[17] DAI T, CAI J, ZHANG Y, et al. Second-order attention network for single image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 11065-11074.
[18] LIU J, ZHANG W, TANG Y, et al. Residual feature aggregation network for image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 2359-2368.
[19] KIM J, LEE J K, LEE K M. Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1637-1645.
[20] TAI Y, YANG J, LIU X. Image super-resolution via deep recursive residual network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,2017: 3147-3155.
[21] HUI Z, GAO X, YANG Y, et al. Lightweight image super-resolution with information multi-distillation network[C]// Proceedings of the 27th ACM International Conference on Multimedia, 2019: 2024-2032.
[22] LEI S, SHI Z, ZOU Z. Super-resolution for remote sensing images via local-global combined network[J]. IEEE Geoscience and Remote Sensing Letters, 2017 (8): 1243-1247.
[23] PAN Z, MA W, GUO J, et al. Super-resolution of single remote sensing image based on residual dense backprojection networks[J]. IEEE Transactions on Geoscience and Remote Sensing, 2019 (10): 7918-7933.
[24] DONG X, WANG L, SUN X, et al. Remote sensing image super-resolution using second-order multi-scale networks [J]. IEEE Transactions on Geoscience and Remote Sensing, 2020 (4): 3473-3485.
[25] ZHANG D, SHAO J, LI X, et al. Remote sensing image super-resolution via mixed high-order attention network [J]. IEEE Transactions on Geoscience and Remote Sensing, 2020 (6): 5183-5196.
[26] ZHANG H, WANG P, JIANG Z. Nonpairwise-trained cycle convolutional neural network for single remote sensing image super-resolution[J]. IEEE Transactions on Geoscience and Remote Sensing, 2020 (5): 4250-4261.
[27] LU Z, LI J, LIU H, et al. Transformer for single image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 457-466.
[28] SHI W, CABALLERO J, HUSZAR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1874-1883.
[29] 吕佳, 许鹏程. 多尺度自适应上采样的图像超分辨率重建算法[J]. 计算机科学与探索, 2023, 17(4):879-891.
LYU J, XU P C. Multi-scale adaptive upsampling image super-resolution reconstruction algorithm[J]. Journal of Frontiers of Computer Science and Technology, 2023,17 (4): 879-897.
[30] WANG Z, LI L, XUE Y, et al. FeNet: feature enhancement network for lightweight remote-sensing image super-resolution [J]. IEEE Transactions on Geoscience and Remote Sensing, 2022 (60): 1-12.
[31] TIMOFTE R, AGUSTSSON E, VAN GOOL L, et al. Ntire 2017 challenge on single image super-resolution: methods and results[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017: 114-125.
[32] YANG Y, NEWSAM S. Bag-of-visual-words and spatial extensions for land-use classification[C]//Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems, 2010: 270-279.
[33] BEVILACQUA M, ROUMY A, GUILLEMOT C, et al. Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]//Proceedings of the British Machine Vision Conference (BMVC), 2012: 1-10.
[34] YANG J, WRIGHT J, HUANG T S, et al. Image super-resolution via sparse representation[J]. IEEE Transactions on Image Processing, 2010 (11): 2861-2873.
[35] MARTIN D, FOWLKES C, TAL D, et al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics [C]//Proceedings of the Eighth IEEE International Conference on Computer Vision(ICCV 2001), 2001: 416-423.
[36] HUANG J B, SINGH A, AHUJA N. Single image super-resolution from transformed self-exemplars[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 5197-5206.
[37] AHN N, KANG B, SOHN K A. Fast, accurate, and lightweight super-resolution with cascading residual network[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 252-268.
[38] TIAN C, ZHUGE R, WU Z, et al. Lightweight image super-resolution with enhanced CNN[J]. Knowledge-Based Systems, 2020,205: 106235.
[39] ZOU W, YE T, ZHENG W, et al. Self-calibrated efficient transformer for lightweight super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 930-939.
[40] LAN R, SUN L, LIU Z, et al. MADNet: a fast and lightweight network for single-image super resolution[J]. IEEE Transactions on Cybernetics, 2020 (3): 1443-1453.