Review of Single Image Super-Resolution Based on Deep Learning

doi:10.3778/j.issn.1002-8331.2106-0055

Abstract

Abstract:

Image Super-Resolution（SR） is a technique that uses signal processing and machine learning to reconstruct High Resolution（HR） images from single or multiple Low Resolution（LR） images. Due to the unpredictability of sub-pixel displacement between multiple LR images, Single Image Super-Resolution（SISR） has gradually become the main direction of super-resolution research. In recent years, deep learning method has been developed rapidly and widely used in the field of image processing. Therefore, the deep learning algorithms and network models used in SISR are summarized systematically. Firstly, the setting and evaluation index of SR are introduced. Then, the deep learning algorithms of SISR reconstruction are discussed and compared, mainly from three aspects of network structure design, loss function and upsampling method. Next, the commonly used standard data sets are introduced, and several typical algorithms based on different network models are selected for experimental comparative analysis. Finally, the future research trend and development direction of SR technology are prospected.

Key words: image super-resolution, deep learning, Convolutional Neural Network（CNN）, image processing

摘要：

图像超分辨重建（Super-Resolution，SR）是指利用信号处理和机器学习等方法，从单幅或者多幅低分辨率图像（Low Resolution，LR）中重建对应的高分辨率图像（High Resolution，HR）的技术。由于多幅LR图像之间亚像素位移的不可预知性，单幅图像超分辨重建（Single Image Super-Resolution，SISR）逐渐成为超分辨研究的主要方向。近年来，深度学习方法得到迅速发展，并广泛应用到图像处理领域。因此，针对单幅图像超分辨重建所使用的深度学习相关算法和网络模型进行系统的总结。介绍图像超分辨问题的设置和评价指标；讨论和比较单幅图像超分辨重建的深度学习算法，主要从网络结构设计、损失函数和上采样方式三方面进行论述；介绍常用的标准数据集，并选用基于不同网络模型的几种典型算法进行实验对比分析；展望图像超分辨技术未来的研究趋势和发展方向。

关键词: 图像超分辨, 深度学习, 卷积神经网络（CNN）, 图像处理

ZHANG De, LIN Qingyu, GUO Maozu. Review of Single Image Super-Resolution Based on Deep Learning[J]. Computer Engineering and Applications, 2021, 57(22): 28-41.

张德，林青宇，郭茂祖. 单幅图像超分辨重建的深度学习方法综述[J]. 计算机工程与应用, 2021, 57(22): 28-41.

References

[1] ISAAC J S，KULKARNI R.Super resolution techniques for medical image processing[C]//Proceedings of International Conference on Technologies for Sustainable Development，2015：2170-2175.
[2] 邢晓羊，魏敏，符颖.基于特征损失的医学图像超分辨率重建[J].计算机工程与应用，2018，54（20）：202-207.
XING X Y，WEI M，FU Y.Super-resolution reconstruction of medical images using feature-based loss[J].Computer Engineering and Applications，2018，54（20）：202-207.
[3] JIA X，LI F，FRASER D.Universal HMT based super resolution for remote sensing images[C]//Proceedings of IEEE International Conference on Image Processing，2008：333-336.
[4] ZHANG L，ZHANG H，SHEN H，et al.A super-resolution reconstruction algorithm for surveillance images[J].Signal Processing，2010，90（3）：848-859.
[5] HARRIS J L.Diffraction and resolving power[J].Journal of the Optical Society of America，1964，54（7）：931-936.
[6] GOODMAN J W.Introduction to Fourier optics[M].2nd ed.New York：McGrawHill，1996.
[7] BROWN H A.Effect of truncation on image enhancement by prelate spheroidal functions[J].Journal of the Optical Society of America，1969，59（2）：228-229.
[8] WADAKA S，SATO T.Super-resolution in incoherent imaging system[J].Journal of the Optical Society of America，1975，65（3）：354-355.
[9] KEYS R.Cubic convolution interpolation for digital image processing[J].IEEE Transactions on Acoustics，Speech，and Signal Processing，1981，29（6）：1153-1160.
[10] KIM H，CHA Y，KIM S.Curvature interpolation method for image zooming[J].IEEE Transactions on Image Processing，2011，20（7）：1895-1903.
[11] MISHIBA K，SUZUKI T，IKEHARA M.Edge-adaptive image interpolation using constrained least squares[C]//Proceedings of IEEE International Conference on Image Processing，2010：2837-2840.
[12] HAN J W，KIM J H，SULL S，et al.New edge-adaptive image interpolation using anisotropic Gaussian filters[J].Digital Signal Processing，2013，23（1）：110-117.
[13] LIU Y，ZHANG Y，GUO Q，et al.Image interpolation based on weighted and blended rational function[C]//Asian Conference on Computer Vision，2014：78-88.
[14] IRANI M，PELEG S.Improving resolution by image registration[J].Graphical Models and Image Processing，1991，53（3）：231-239.
[15] YANG X，ZHANG Y，ZHOU D，et al.An improved iterative back projection algorithm based on ringing artifacts suppression[J].Neurocomputing，2015，162：171-179.
[16] TAN J，TAO Z，CAO A，et al.An edge-preserving iterative back-projection method for image super-resolution[C]//Proceedings of International Conference on Digital Image Processing，2016：1-8.
[17] TEKALP A M，OZKAN M K，SEZAN M I.High-resolution image reconstruction from lower-resolution image sequences and space-varying image restoration[C]//Proceedings of IEEE International Conference on Acoustics，Speech and Signal Processing，1992，3：169-172.
[18] PATTI A J，ALTUNBASAK Y.Artifact reduction for set theoretic super resolution image reconstruction with edge adaptive constraints and higher-order interpolants[J].IEEE Transactions on Image Processing，2001，10（1）：179-186.
[19] KIM J Y，PARK R H，YANG S.Super-resolution using POCS-based reconstruction with artifact reduction constraints[C]//Proceedings of SPIE on Visual Communications and Image Processing，2005，5960（4）：1810-1818.
[20] SUN D，GAO D，LU Y，et al.A high quality single-image super-resolution algorithm based on linear Bayesian MAP estimation with sparsity prior[J].Digital Signal Processing，2014，35：45-52.
[21] VRIGKAS M，NIKOU C，KONDI L P.Robust maximum a posteriori image super-resolution[J].Journal of Electronic Imaging，2014，23（4）：30-35.
[22] CHANTAS G K，GALATSANOS N P，WOODS N A.Super-resolution based on fast registration and maximum a posteriori reconstruction[J].IEEE Transactions on Image Processing，2007，16（7）：1821-1830.
[23] CHANG H，YEUNG D Y，XIONG Y.Super-resolution through neighbor embedding[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2004：1-8.
[24] BEVILACQUA M，ROUMY A，GUILLEMOT C，et al.Neighbor embedding based single-image super-resolution using semi-nonnegative matrix factorization[C]//Proceedings of IEEE International Conference on Acoustics，Speech and Signal Processing，2012：1289-1292.
[25] CHEN X，QI C.Low-rank neighbor embedding for single image super-resolution[J].IEEE Signal Processing Letters，2014，21（1）：79-82.
[26] YANG J，WRIGHT J，HUANG T S，et al.Image super-resolution via sparse representation[J].IEEE Transactions on Image Processing，2010，19（11）：2861-2873.
[27] DONG W，ZHANG L，SHI G，et al.Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization[J].IEEE Transactions on Image Processing，2011，20（7）：1838-1857.
[28] PELEG T，ELAD M.A statistical prediction model based on sparse representations for single image super-resolution[J].IEEE Transactions on Image Processing，2014，23（6）：2569-2582.
[29] NI K S，NGUYEN T Q.Image super-resolution using support vector regression[J].IEEE Transactions on Image Processing，2007，16（6）：1596-1610.
[30] KIM K I，KWON Y.Single-image super-resolution using sparse regression and natural image prior[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2010，32（6）：1127-1133.
[31] HE H，SIU W C.Single image super-resolution using Gaussian process regression[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recog-
nition，2011：449-456.
[32] BAKER S，KANADE T.Limits on super-resolution and how to break them[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2002，24（9）：1167-1183.
[33] LI X G，LI F H，ZHUO L.Research progress of joint reconstruction of high resolution and high dynamic range images[J].Measurement and Control Technology，2012，31（5）：8-12.
[34] DONG C，LOY C C，He K，et al.Learning a deep convolutional network for image super-resolution[C]//Proceedings of European Conference on Computer Vision，2014：184-189.
[35] WANG Z W，LIU D，YANG J C，et al.Deep networks for image super-resolution with sparse prior[C]//Proceedings of IEEE International Conference on Computer Vision，2015：370-378.
[36] KIM J，LEE J K，LEE K M.Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2016：1646-1654.
[37] KIM J，LEE J K，LEE K M.Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2016：1637-1645.
[38] LEDIG C，THEIS L，HUSZAR F，et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2017：105-114.
[39] TONG T，LI G，LIU X，et al.Image super-resolution using dense skip connections[C]//Proceedings of IEEE International Conference on Computer Vision，2017：4809-4817.
[40] ZHANG Y L，TIAN Y P，KONG Y，et al.Residual dense network for image super-resolution[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2018：2472-2481.
[41] ZHANG Y L，LI K P，LI K，et al.Image super-resolution using very deep residual channel attention networks[C]//Proceedings of European Conference on Computer Vision，2018：294-310.
[42] DAI T，CAI J R，ZHANG Y B，et al.Second-order attention network for single image super-resolution[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2019：1057-1066.
[43] LI Z，YANG J L，Liu Z，et al.Feedback network for image super-resolution[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2019：3862-3871.
[44] YANG F Z，YANG H，FU J L，et al.Learning texture transformer network for image super-resolution[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2020：5790-5799.
[45] MA C，RAO Y M，CHENG Y，et al.Structure-preserving super resolution with gradient guidance[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2020：7766-7775.
[46] 史振威，雷森.图像超分辨重建算法综述[J].数据采集与处理，2020，35（1）：1-20.
SHI Z W，LEI S.Review of image super-resolution reconstruction[J].Journal of Data Acquisition and Processing，2020，35（1）：1-20.
[47] 唐艳秋，潘泓，朱亚平，等.图像超分辨率重建研究综述[J].电子学报，2020，48（7）：1407-1420.
TANG Y Q，PAN H，ZHU Y P，et al.A survey of image super-resolution reconstruction[J].Acta Electronica Sinica，2020，48（7）：1407-1420.
[48] 刘颖，朱丽，林庆帆，等.图像超分辨技术的回顾与展望[J].计算机科学与探索，2020，14（2）：181-199.
LIU Y，ZHU L，LIN Q F，et al.Review and prospect of image super-resolution technology[J].Journal of Frontiers of Computer Science and Technology，2020，14（2）：181-199.
[49] 南方哲，钱育蓉，行艳妮，等.基于深度学习的单图像超分辨率重建研究综述[J].计算机应用研究，2020，37（2）：321-326.
NAN F Z，QIAN Y R，XING Y N，et al.Survey of single image super resolution based on deep learning[J].Application Research of Computers，2020，37（2）：321-326.
[50] ZHANG K，ZUO W，ZHANG L.Learning a single convolutional super-resolution network for multiple degradations[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2018：3262-3271.
[51] WANG Z，BOVIK A C，SHEIKH H R，et al.Image quality assessment：from error visibility to structural similarity[J].IEEE Transactions on Image Processing，2004，13（4）：600-612.
[52] WANG Z，BOVIK A C.Mean squared error：lot it or leave it? A new look at signal fidelity measures[J].IEEE Signal Processing Magazine，2009，26（1）：98-117.
[53] DONG C，LOY C C，TANG X O.Accelerating the super-resolution convolutional neural network[C]//Proceedings of European Conference on Computer Vision，2016：391-407.
[54] SHI W Z，CABALLERO J，HUSZAR F，et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2016：1874-1883.
[55] LAI W S，HUANG J B，AHUJAN，et al.Deep Laplacian pyramid networks for fast and accurate super-resolution[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2017：624-632.
[56] LIM B，SON S，KIM H，et al.Enhanced deep residual networks for single image super-resolution[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops，2017：1132-1140.
[57] LI J C，FANG F M，MEI K F，et al.Multi-scale residual network for image super-resolution[C]//Proceedings of European Conference on Computer Vision，2018：527-542.
[58] TAI Y，YANG J，LIU X M.Image super-resolution via deep recursive residual network[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2017：2790-2798.
[59] CHOI J，KIM M.A deep convolutional neural network with selection units for super-resolution[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops，2017：1150-1156.
[60] WANG X T，YU K，WU S H，et al.ESRGAN：enhanced super-resolution generative adversarial networks[C]//Proceedings of European Conference on Computer Vision Workshops，2018：63-79.
[61] HE K，ZHANG X，REN S Q，et al.Deep residual learning for image recognition[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2016：770-778.
[62] 应自炉，龙祥.多尺度密集残差网络的单幅图像超分辨率重建[J].中国图象图形学报，2019，24（3）：410-419.
YING Z L，LONG X.Single-image super-resolution construction based on multi-scale dense residual network[J].Journal of Image and Graphics，2019，24（3）：410-419.
[63] 李现国，冯欣欣，李建雄.多尺度残差网络的单幅图像超分辨率重建[J].计算机工程与应用，2021，57（7）：215-221.
LI X G，FENG X X，LI J X.Single image super-resolution reconstruction based on multi-scale residual network[J].Computer Engineering and Applications，2021，57（7）：215-221.
[64] 程德强，郭昕，陈亮亮，等.多通道递归残差网络的图像超分辨率重建[J].中国图象图形学报，2021，26（3）：605-618.
CHENG D Q，GUO X，CHEN L L，et al.Image super-resolution reconstruction from multi-channel recursive residual network[J].Journal of Image and Graphics，2021，26（3）：605-618.
[65] HUANG G，LIU Z，VAN DER MAATEN L，et al.Densely connected convolutional networks[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2017：2261-2269.
[66] 王海勇，张开心，管维正.基于密集Inception的单图像超分辨率重建[J/OL].计算机应用（2021-04-15）[2021-07-13].https：//kns.cnki.net/kcms/detail/51.1307.TP.20210415.1300.
004.html.
WANG H Y，ZHANG K X，GUAN W Z.Single image super-resolution reconstruction based on dense Inception[J/OL].Journal of Computer Applications（2021-04-15）[2021-07-13].https：//kns.cnki.net/kcms/detail/51.1307.TP.
20210415.1300.004.html.
[67] 李彬，王平，赵思逸.基于双重注意力机制的图像超分辨重建算法[J].图学学报，2021，42（2）：206-215.
LI B，WANG P，ZHAO S Y.Image super-resolution reconstruction based on dual attention mechanism[J].Journal of Graphics，2021，42（2）：206-215.
[68] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial nets[C]//Proceedings of Advances in Neural Information Processing Systems，2014：2672-2680.
[69] 彭晏飞，高艺，杜婷婷，等.生成对抗网络的单图像超分辨率重建方法[J].计算机科学与探索，2020，14（9）：1612-1620.
PENG Y F，GAO Y，DU T T，et al.Single image super-resolution reconstruction method for generative adversarial network[J].Journal of Frontiers of Computer Science and Technology，2020，14（9）：1612-1620.
[70] 汪鑫耘，李丹.生成式对抗网络在超分辨率图像重建中的应用[J].计算机科学与探索，2020，14（4）：680-687.
WANG X Y，LI D.Application of generative adversarial network in super-resolution image reconstruction[J].Journal of Frontiers of Computer Science and Technology，2020，14（4）：680-687.
[71] JUSTIN J，ALEXANDRE A，LI F F.Perceptual losses for real-time style transfer and super-resolution[C]//Proceedings of European Conference on Computer Vision，2016：694-711.
[72] ALEXEY D，THOMAS B.Generating images with perceptual similarity metrics based on deep networks[C]//Proceedings of Annual Conference on Neural Information Processing Systems，2016：658-666.
[73] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[C]//Proceedings of International Conference on Learning Representations，2015：234-240.
[74] LEON A G，ALEXANDER S E，MATTHIAS B.Texture synthesis using convolutional neural networks[C]//Proceedings of Annual Conference on Neural Information Processing Systems，2015：262-270.
[75] LEON A G，ALEXANDER S E，MATTHIAS B.Image style transfer using convolutional neural networks[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2016：2414-2423.
[76] SAJJADI M，SCHOLKOPF B，HIRSCH M，et al.Enhance-
Net：single image super-resolution through automated texture synthesis[C]//Proceedings of IEEE International Conference on Computer Vision，2017：4501-4510.
[77] WANG Y F，PERAZZI F，MCWILLIAMS B，et al.A fully progressive approach to single-image super-resolution[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops，2018：977-986.
[78] YUAN Y，LIU S Y，ZHANG J W，et al.Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops，2018：814-823.
[79] MAO X D，LI Q，XIE H R，et al.Least squares generative adversarial networks[C]//Proceedings of IEEE International Conference on Computer Vision，2017：2813-2821.
[80] ALEXIA J M.The relativistic discriminator：a key element missing from standard GAN[C]//Proceedings of International Conference on Learning Representations，2019：1240-1246.
[81] ISHAAN G，FARUK A，MARTIN A，et al.Improved training of Wasserstein GANs[C]//Proceedings of Annual Conference on Neural Information Processing Systems，2017：5768-5778.
[82] BEVILACQUA M，ROUMY A，GUILLEMOT C，et al.Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]//Proceedings of British Machine Vision Conference，2012：201-210.
[83] ZEYDE R，ELAD M，PROTTER M.On single image scale-up using sparse-representations[C]//Proceedings of International Conference on Curves and Surfaces，2010：711-730.
[84] MARTIN D，FOWLKES C，TAL D，et al.A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings of IEEE International Conference on Computer Vision，2001：416-423.
[85] HUANG J B，SINGH A，AHUJA N.Single image super-resolution from transformed self-exemplars[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2015：5197-5206.
[86] FUJIMOTO A，OGAWA T，YAMAMOTO K，et al.Manga-
109 dataset and creation of metadata[C]//Proceedings of International Workshop on Comics Analysis，Processing and Understanding，2016：1110-1118.
[87] AGUSTSSON E，TIMOFTE R.NTIRE 2017 challenge on single image super-resolution：dataset and study[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops，2017：1122-1131.
[88] TIMOFTE R，AGUSTSSON E，GOOL L C，et al.NTIRE 2017 challenge on single image super-resolution：methods and results[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops，2017：1110-1121.
[89] RUSSAKOVSKY O，DENG J，SU H，et al.Image net large scale visual recognition challenge[J].International Journal of Computer Vision，2015，115（3）：211-252.
[90] ZHANG K，VAN G L，TIMOFTE R.Deep unfolding network for image super-resolution[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2020：3214-3223.