Review of Research on Lightweight Image Super-Resolution

doi:10.3778/j.issn.1002-8331.2403-0230

Abstract

Abstract: In recent years, image super-resolution (SR) based on deep learning has received widespread attention. The purpose of image SR is to improve the resolution of images to facilitate further processing of images, such as target detection, image classification and face recognition, etc. The research on image SR has achieved rapid development in recent years, but there are still few related reviews on lightweight SR models. By analyzing the current research status of lightweight SR methods which are based on deep learning and loss function, a new classification of current lightweight SR models is made, which are traditional convolution methods and attention mechanism methods. The development history and latest progress of lightweight SR methods for images are systematically given, the advantages and disadvantages of each method are pointed out. Finally, by analyzing the existing problems of current lightweight SR technology, the future research directions of lightweight image SR method are given.

Key words: image super-resolution, lightweight, deep learning, convolutional neural network, attention mechanism

摘要： 基于深度学习的图像超分辨率（super-resolution，SR）受到广泛关注，其目的是提高图像的分辨率，以便对图像做进一步的处理，如目标检测、图像分类和人脸识别等。图像SR领域相关研究近年来取得了迅猛发展，但有关轻量级SR模型的相关综述还不多见。对基于深度学习的轻量级SR方法研究现状和损失函数进行了分析，并对目前轻量级SR方法进行了新的分类，分别为传统卷积方法和注意力机制方法。系统梳理了图像轻量级SR方法的发展历程和最新进展，指出了每一种方法存在的优势和缺陷。最后对当前轻量级SR技术存在的问题进行了分析，并给出了轻量级图像SR方法未来的研究方向。

关键词: 图像超分辨率, 轻量级, 深度学习, 卷积神经网络, 注意力机制

ZHU Xinfeng, SONG Jian. Review of Research on Lightweight Image Super-Resolution[J]. Computer Engineering and Applications, 2024, 60(16): 49-60.

朱新峰, 宋健. 轻量级图像超分辨率研究综述[J]. 计算机工程与应用, 2024, 60(16): 49-60.

References

[1] 李云红, 王珍, 张凯兵, 等. 基于学习的图像超分辨重建方法综述[J]. 计算机工程与应用, 2018, 54(15): 13-21.
LI Y H, WANG Z, ZHANG K B, et al. Survey on example learning-based single image super-resolution technique[J]. Computer Engineering and Applications, 2018, 54(15): 13-21.
[2] 孙旭, 李晓光, 李嘉锋, 等. 基于深度学习的图像超分辨率复原研究进展[J]. 自动化学报, 2017, 43(5): 697-709.
SUN X, LI X G, LI J F, et al. Review on deep learning based image super-resolution restoration algorithms[J] Acta Automatica Sinica, 2017, 43(5): 697-709.
[3] DONG C, LOY C C, HE K, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307.
[4] YANG W M, ZHANG X C, TIAN Y P, et al. Deep learning for single image super-resolution: a brief review[J]. IEEE Transactions on Multimedia, 2019, 21(12): 3106-3121.
[5] CHEN H G, HE X H, QING L B, et al. Real-world single image super-resolution: a brief review[J]. Information Fusion, 2022, 79: 124-145.
[6] JIANG J J, WANG C Y, LIU X M, et al. Deep learning-based face super-resolution: a survey[J]. ACM Computing Surveys, 2023, 55(1): 13.
[7] LIU A R, LIU Y H, GU J J, et al. Blind image super-resolution: a survey and beyond[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(5): 5461-5480.
[8] 杨才东, 李承阳, 李忠博, 等. 深度学习的图像超分辨率重建技术综述[J]. 计算机科学与探索, 2022, 16(9): 1990-2010.
YANG C D, LI C Y, LI Z B, et al. Review of image super-resolution reconstruction algorithms based on deep learning[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(9): 1990-2010.
[9] GENDY G, HE G, SABOR N. Lightweight image super-resolution based on deep learning: state-of-the-art and future directions[J]. Information Fusion, 2023, 94: 284-310.
[10] DONG C, LOY C C, TANG X. Accelerating the super-resolution convolutional neural network[C]//Proceedings of the 14th European Conference on Computer Vision, Amsterdam, Oct 11-14, 2016. Cham: Springer, 2016: 391-407.
[11] SHI W, CABALLERO J, HUSZAR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1874-1883.
[12] KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1646-1654.
[13] KIM J, LI J K, LI K K M. Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1637-1645.
[14] LI H L, LIU Z H, LIU Y, et al. Lightweight image super-resolution network using 3D convolutional neural networks[J]. Journal of Electronic Imaging, 2024, 33(1): 013016.
[15] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
[16] LIM B, SON S, KIM H, et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017: 136-144.
[17] YING T, JIAN Y, XIAOMING L. Image super-resolution via deep recursive residual network[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 2790-2798.
[18] AHN N, KANG B, SOHN K A. Fast, accurate, and lightweight super-resolution with cascading residual network[C]//Proceedings of the 15th European Conference on Computer Vision, 2018: 252-268.
[19] ZHANG Y, TIAN Y, KONG Y, et al. Residual dense network for image restoration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(7): 2480-2495.
[20] SUN B, ZHANG Y, JIANG S, et al. Hybrid pixel-unshuffled network for lightweight image super?resolution[C]//Proceedings of the 37th AAAI Conference on Artificial Intelligence, 2023: 2375-2383.
[21] ZHANG Y, BAI H M, BING Y X, et al. Lightweight image super-resolution with ConvNeXt residual network[J]. Neural Processing Letters, 2023, 55(7): 9545-9561.
[22] GAO H, ZHUANG L, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 2261-2269.
[23] TONG T, LI G, LIU X, et al. Image super-resolution using dense skip connections[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision, 2017: 4799-4807.
[24] FENG Z, LAI J, XIE X, et al. Image super-resolution via a densely connected recursive network[J]. Neurocomputing, 2018, 316: 270-276.
[25] ANWAR S, BARNES N. Densely residual laplacian super-resolution[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(3): 1192-1204.
[26] LV X, WANG C Z, FAN X D, et al. A novel image super-resolution algorithm based on multi-scale dense recursive fusion network[J]. Neurocomputing, 2022, 489: 98-111.
[27] HINTON G, VINYALS O, DEAN J. Distilling the knowledge in a neural network[J]. arXiv:1503.02531, 2015.
[28] GOU J, YU B, MAYBANK S J, et al. Knowledge distillation: a survey[J]. International Journal of Computer Vision, 2021, 129(6): 1789-1819.
[29] ZHENG H, XIUMEI W, XINBO G. Fast and accurate single image super-resolution via information distillation network[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018: 723-731.
[30] YANG X, GUO Y, LI Z, et al. MRDN: a lightweight multi-stage residual distillation network for image super-resolution[J]. Expert Systems with Applications, 2022, 204: 117594.
[31] TIAN C, YUAN Y, ZHANG S, et al. Image super-resolution with an enhanced group convolutional neural network[J]. Neural Networks, 2022, 153: 373-385.
[32] GAO X, ZHOU Y, WU S N, et al. Residual multi-branch distillation network for efficient image super-resolution[J]. Multimedia Tools and Applications, 2024. DOI:10.1007/s11042-024-18504-5.
[33] GAO Q, ZHAO Y, LI G, et al. Image super-resolution using knowledge distillation[C]//Proceedings of the 2018 Asian Conference on Computer Vision. Cham: Springer, 2018: 527-541.
[34] HE Z, DAI T, LU J, et al. FAKD: feature-affinity based knowledge distillation for efficient image super-resolution[C]//Proceedings of the 2020 IEEE International Conference on Image Processing, 2020: 518-522.
[35] WANG C, CHEN D, MEI J P, et al. SemCKD: semantic calibration for cross-layer knowledge distillation[J]. IEEE Transactions on Knowledge and Data Engineering, 2023, 35(6): 6305-6319.
[36] FANG H X, LONG Y W, HU X Y, et al. Dual cross knowledge distillation for image super-resolution[J]. Journal of Visual Communication and Image Representation, 2023, 95: 103858.
[37] FAN Y C, SHI H H, YU J H, et al. Balanced two-stage residual networks for image super-resolution[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition: Workshops, 2017: 1157-1164.
[38] LIU H, CAO F L, WEN C L, et al. Lightweight multi-scale residual networks with attention for image super-resolution[J]. Knowledge-Based Systems, 2020, 203: 106103.
[39] ZONG Z, ZHA L, JIANG J, et al. Asymmetric information distillation network for lightweight super resolution[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 1249-1258.
[40] HU J, SHEN L, ALBANIE S, et al. Squeeze-and-excitation networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(8): 2011-2023.
[41] ZHANG Y L, LI K P, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]//LNCS 11211: Proceedings of the 15th European Conference on Computer Vision, 2018: 294-310.
[42] YANG A P, YANG B W, JI Z, et al. Lightweight group convolutional network for single image super-resolution[J]. Information Sciences, 2020, 516: 220-233.
[43] PENG C M, SHU P, HUANG X Y, et al. LCRCA: image super-resolution using lightweight concatenated residual channel attention networks[J]. Applied Intelligence, 2022, 52(9): 10045-10059.
[44] LU J, LUO X. Image super-resolution with enhanced channel attention residual network[C]//Proceedings of the 2023 8th International Conference on Intelligent Computing and Signal Processing, 2023: 16501-16504.
[45] YASIR M, ULLAH I, CHOI C. Depthwise channel attention network (DWCAN): an efficient and lightweight model for single image super-resolution and metaverse gaming[J]. Expert Systems, 2024, 41(4): e13516.
[46] SOH J W, CHO N I. Lightweight single image super-resolution with multi-scale spatial attention networks[J]. IEEE Access, 2020, 8: 35383-35391.
[47] CAI Y X, LAI H C, JIA Z H, et al. Lightweight spatial-channel adaptive coordination of multilevel refinement enhancement network for image reconstruction[J]. Knowledge-Based Systems, 2022, 256: 109824.
[48] CHEN J C, WANG W L, XING F S, et al. Multi-feature fusion attention network for single image super-resolution[J]. IET Image Processing, 2023, 17(5): 1389-1402.
[49] TOMAR A S, ARYA K V, RAJPUT S S. Noise robust face super-resolution via learning of spatial attentive features[J]. Multimedia Tools and Applications, 2023, 82(16): 25449-25465.
[50] ZHAO H, KONG X, HE J, et al. Efficient image super-resolution using pixel attention[C]//Proceedings of the 16th European Conference on Computer Vision Workshops, Glasgow, Aug 23-28, 2020. Cham: Springer, 2020: 56-72.
[51] ZHAO K H, LU T, ZHANG Y D, et al. Face super-resolution via triple-attention feature fusion network[J]. IEICE Transactions on Fundamentals of Electronics Communications and Computer Sciences, 2022, E105A(4): 748-752.
[52] WANG X, ZHANG S F, LIN Y Y, et al. Pixel attention convolutional network for image super-resolution[J]. Neural Computing & Applications, 2023, 35(11): 8589-8599.
[53] JIANG L F，ZHONG M Z，QIU F C. Single-image super-resolution based on a self-attention deep neural network[C]//Proceedings of the 2020 13th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics, 2020: 387-391.
[54] LIU Y H, CHU Z Z, LI B. A local and non-local features based feedback network on super-resolution[J]. Sensors, 2022, 22(24): 9604.
[55] ZENG K, LIN H J, YAN Z Q, et al. Densely connected transformer with linear self-attention for lightweight image super-resolution[J]. IEEE Transactions on Instrumentation and Measurement, 2023, 72: 1-12.
[56] PARK K, SOH J W, CHO N I. A dynamic residual self-attention network for lightweight single image super-resolution[J]. IEEE Transactions on Multimedia, 2023, 25: 907-918.
[57] HAN X W, WANG L, WANG X P, et al. A multi-scale recursive attention feature fusion network for image super-resolution reconstruction algorithm[J]. Sensors, 2023, 23(23): 9458.
[58] LI H A, WANG D, ZHANG J, et al. Image super-resolution reconstruction based on multi-scale dual-attention[J]. Connection Science, 2023, 35(1): 2182487.
[59] XIE F, LU P, LIU X Y. Multi-scale convolutional attention network for lightweight image super-resolution[J]. Journal of Visual Communication and Image Representation, 2023, 95: 103889.
[60] WU H P, GUI J, ZHANG J, et al. Pyramidal dense attention networks for single image super-resolution[J]. IET Image Processing, 2022, 16(12): 3247-3257.
[61] GENDY G, SABOR N, HOU J C, et al. Balanced spatial feature distillation and pyramid attention network for lightweight image super-resolution[J]. Neurocomputing, 2022, 509: 157-166.
[62] CHEN S, BI X P, ZHANG L F. Fused pyramid attention network for single image super-resolution[J]. IET Image Processing, 2023, 17(6): 1681-1693.
[63] MEI Y Q, FAN Y C, ZHANG Y L, et al. Pyramid attention network for image restoration[J]. International Journal of Computer Vision, 2023, 131(12): 3207-3225.
[64] LAI W S, HUANG J B, AHUJA N, et al. Fast and accurate image super-resolution with deep laplacian pyramid networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(11): 2599-2613.
[65] WU Q, FAN C X, LI Y, et al. A novel perceptual loss function for single image super-resolution[J]. Multimedia Tools and Applications, 2020, 79(29/30): 21265-21278.
[66] JIANG D H, SAI Z, LEI D, et al. Multi-scale generative adversarial network for image super-resolution[J]. Soft Computing, 2022, 26(8): 3631-3641.
[67] 吴靖, 叶晓晶, 黄峰, 等. 基于深度学习的单帧图像超分辨率重建综述[J]. 电子学报, 2022, 50(9): 2265-2294.
WU J, YE X J, HUANG F, et al. A review of single image super-resolution reconstruction based on deep learning[J]. Acta Electronica Sinica, 2022, 50(9): 2265-2294.
[68] RADOSAVOVIC I, KOSARAJU R P, GIRSHICK R, et al. Designing network design spaces[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 10428-10436.