Survey on Deep Learning Based Image Super-Resolution

doi:10.3778/j.issn.1002-8331.2105-0418

Abstract

Abstract:

Image super-resolution reconstruction is the process of using low-resolution images to reconstruct the corresponding high-resolution images. At present, image super-resolution technology has been successfully applied in the fields of computer vision and image processing. In recent years, due to deep learning’s ability of self-learning from a large amount of data, it has been widely used in the field of image super-resolution. This article introduces the background of image super-resolution reconstruction, and summarizes the deep learning based image super-resolution model in detail, and then elaborates the image super-resolution technology in satellite remote sensing images, medical imaging, video surveillance, and industrial inspection tasks application. Finally, this article summarizes the current research status and future development directions of image super-resolution algorithms.

Key words: image super-resolution, deep learning, convolutional neural network, generative adversarial network

摘要：

图像超分辨率重建是用低分辨率图像重建出对应的高分辨率图像的过程。目前，图像超分辨率技术已经成功应用于计算机视觉和图像处理领域。近年来，由于深度学习具有能够从大量数据中自动学习特征的能力，因此被广泛应用于图像超分辨率领域中。介绍了图像超分辨重建的背景，详细总结了用于图像超分辨率的深度学习模型，阐述了图像超分辨率技术在卫星遥感图像、医学影像、视频监控、工业检测任务方面的应用。总结了图像超分辨算法的当前研究现状以及未来发展方向。

关键词: 图像超分辨率, 深度学习, 卷积神经网络, 生成对抗网络

XIA Hao, LYU Hongfeng, LUO Jun, CAI Nian. Survey on Deep Learning Based Image Super-Resolution[J]. Computer Engineering and Applications, 2021, 57(24): 51-60.

夏皓，吕宏峰，罗军，蔡念. 图像超分辨率深度学习研究及应用进展[J]. 计算机工程与应用, 2021, 57(24): 51-60.

References

[1] ZHANG L，NIE J，WEI W，et al.Deep blind hyperspectral image super-resolution[J].IEEE Transactions on Neural Networks and Learning Systems，2020（99）：1-13.
[2] MASUTANI E M，BAHRAMI N，HSIAO A.Deep learning single-frame and multiframe super-resolution for cardiac MRI[J].Radiology，2020，295（3）：552-561.
[3] WANG M，CHEN Z，WU Q M J，et al.Improved face super-resolution generative adversarial networks[J].Machine Vision and Applications，2020，31（4）：1-12.
[4] DONG C，LOY C C，HE K，et al.Learning a deep convolutional network for image super-resolution[C]//European Conference on Computer Vision.Cham：Springer，2014：184-199.
[5] DONG C，LOY C C，HE K，et al.Image super-resolution using deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2015，38（2）：295-307.
[6] DONG C，LOY C C，TANG X.Accelerating the super-resolution convolutional neural network[C]//European Conference on Computer Vision.Cham：Springer，2016：391-407.
[7] SHI W，CABALLERO J，HUSZáR F，et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：1874-1883.
[8] ZHANG K，ZUO W，ZHANG L.Learning a single convolutional super-resolution network for multiple degradations[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：3262-3271.
[9] KIM J，KWON LEE J，MU LEE K.Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：1646-1654.
[10] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409.
1556，2014.
[11] LAI W S，HUANG J B，AHUJA N，et al.Deep laplacian pyramid networks for fast and accurate super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：624-632.
[12] LAI W S，HUANG J B，NARENDRA A，et al.Fast and accurate image super-resolution with deep laplacian pyramid networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2019，41（11）.
[13] LIM B，SON S，KIM H，et al.Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops，2017：136-144.
[14] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[15] FENG X，LI X，LI J.Multi-scale fractal residual network for image super-resolution[J].Applied Intelligence，2021，51（4）：1845-1856.
[16] LAN R，SUN L，LIU Z，et al.Cascading and enhanced residual networks for accurate single-image super-resolution[J].IEEE Transactions on Cybernetics，2020，51（1）：115-125.
[17] 席志红，侯彩燕，袁昆鹏，等.基于深层残差网络的加速图像超分辨率重建[J].光学学报，2019，39（2）：81-90.
XI Z H，HOU C Y，YUAN K P，et al.Super-resolution reconstruction of accelerated image based on deep residual network[J].Acta Optica Sinica，2019，39（2）：81-90.
[18] KIM J，KWON LEE J，MU LEE K.Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：1637-1645.
[19] TAI Y，YANG J，LIU X.Image super-resolution via deep recursive residual network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：3147-3155.
[20] 吴磊，吕国强，薛治天，等.基于多尺度递归网络的图像超分辨率重建[J].光学学报，2019，39（6）：82-89.
WU L，LYU G Q，XUE Z T，et al.Super-resolution reconstruction of images based on multi-scale recursive network[J].Acta Optica Sinica，2019，39（6）：82-89.
[21] JIANG K，WANG Z，YI P，et al.Hierarchical dense recursive network for image super-resolution[J].Pattern Recognition，2020，107：107475.
[22] LIN D，XU G，XU W，et al.SCRSR：an efficient recursive convolutional neural network for fast and accurate image super-resolution[J].Neurocomputing，2020，398：399-407.
[23] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial nets[C]//Advances in Neural Information Processing Systems，2014：2672-2680.
[24] RADFORD A，METZ L，CHINTALA S.Unsupervised representation learning with deep convolutional generative adversarial networks[J].arXiv：1511.06434，2015.
[25] LEDIG C，THEIS L，HUSZáR F，et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：4681-4690.
[26] WANG X，YU K，WU S，et al.Esrgan：enhanced super-resolution generative adversarial networks[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：63-79.
[27] JOLICOEUR-MARTINEAU A.The relativistic discriminator：a key element missing from standard GAN[J].arXiv：1807.00734，2018.
[28] GAO H，CHEN Z，HUANG B，et al.Image super-resolution based on conditional generative adversarial network[J].IET Image Processing，2020，14（13）：3006-3013.
[29] MA T，TIAN W.Back-projection-based progressive growing generative adversarial network for single image super-resolution[J].The Visual Computer，2021，37（5）：925-938.
[30] 彭晏飞，高艺，杜婷婷，等.生成对抗网络的单图像超分辨率重建方法[J].计算机科学与探索，2020，14（9）：1612-1620.
PENG Y F，GAO Y，DU T T，et al.Single image super-resolution reconstruction method for generative adversarial network[J].Journal of Frontiers of Computer Sicence and Technology，2020，14（9）：1612-1620.
[31] ZHOU D W.Single image super-resolution reconstruction based on multi-scale feature mapping adversarial network[J].Signal Processing，2020，166：107251.
[32] GAO H，ZHUANG L.Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：4700-4708.
[33] ZHOU Y，ZHANG Y，XIE X，et al.Image super-resolution based on dense convolutional auto-encoder blocks[J].Neurocomputing，2021，423：98-109.
[34] LIU Z，HUANG J，ZHU C，et al.Residual attention network using multi-channel dense connections for image super-resolution[J].Applied Intelligence，2021，51（1）：85-99.
[35] MA J，WANG X，JIANG J.Image super resolution via dense discriminative network[J].IEEE Transactions on Industrial Electronics，2019，67（7）：5687-5695.
[36] ZHANG X Y，GAO P.Accurate and efficient image super-resolution via global-local adjusting dense network[J].IEEE Transactions on Multimedia，2021，23：1924-1937.
[37] SONG D H，XU C.Efficient residual dense block search for image super-resolution[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2020，34（7）：12007-12014.
[38] LI J C，FANG F M.Multi-scale dense cross network for image super-resolution[J].IEEE Transactions on Circuits and Systems for Video Technology，2021，31（7）：2547-2561.
[39] GU J，LU H，ZUO W，et al.Blind super-resolution with iterative kernel correction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：1604-1613.
[40] WANG Y，WANG L，WANG H，et al.Blind single image super-resolution with a mixture of deep networks[J].Pattern Recognition，2020，102：107169.
[41] LIU P，ZHANG H，CAO Y，et al.Learning cascaded convolutional networks for blind single image super-resolution[J].Neurocomputing，2020，417：371-383.
[42] WANG L，WANG Y，DONG X，et al.Unsupervised degradation representation learning for blind super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：10581-10590.
[43] KIM S Y，SIM H，KIM M.Koalanet：blind super-resolution using kernel-oriented adaptive local adjustment[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：10611-10620.
[44] HUI Z，LI J，WANG X，et al.Learning the non-differentiable optimization for blind super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：2093-2102.
[45] SHEN H，LIN L，LI J，et al.A residual convolutional neural network for polarimetric SAR image super-resolution[J].ISPRS Journal of Photogrammetry and Remote Sensing，2020，161：90-108.
[46] ZHANG D，SHAO J，LI X，et al.Remote sensing image super-resolution via mixed high-order attention network[J].IEEE Transactions on Geoscience and Remote Sensing，2020（99）：1-14.
[47] DONG X，SUN X，JIA X，et al.Remote sensing image super-resolution using novel dense-sampling networks[J].IEEE Transactions on Geoscience and Remote Sensing，2020（99）：1-16.
[48] LI J，CUI R，LI B，et al.Hyperspectral image super-resolution by band attention through adversarial learning[J].IEEE Transactions on Geoscience and Remote Sensing，2020，58（6）：4304-4318.
[49] GUO D，XIA Y，XU L，et al.Remote sensing image super-resolution using cascade generative adversarial nets[J].Neurocomputing，2021，443：117-130.
[50] ZHAO X，ZHANG Y，ZHANG T，et al.Channel splitting network for single MR image super-resolution[J].IEEE Transactions on Image Processing，2019，28（11）：5649-5662.
[51] CHEN L，YANG X，JEON G，et al.A trusted medical image super-resolution method based on feedback adaptive weighted dense network[J].Artificial Intelligence in Medicine，2020，106：101857.
[52] XU L，ZENG X，HUANG Z，et al.Low-dose chest X-ray image super-resolution using generative adversarial nets with spectral normalization[J].Biomedical Signal Processing and Control，2020，55：10160.
[53] SONG T A，CHOWDHURY S R，YANG F，et al.PET image super-resolution using generative adversarial networks[J].Neural Networks，2020，125：83-91.
[54] XIA H，CAI N，WANG H，et al.Brain MR image super-resolution via a deep convolutional neural network with multi-unit upsampling learning[J].Signal，Image and Video Processing，2021，15（12）：1-9.
[55] HE J，ZHENG J，SHEN Y，et al.Facial image synthesis and super-resolution with stacked generative adversarial network[J].Neurocomputing，2020，402：359-365.
[56] 金炜，陈莹.多尺度残差通道注意机制下的人脸超分辨率网络[J].计算机辅助设计与图形学学报，2020，32（6）：959-970.
JIN W，CHEN Y.Multi-scale residual channel attention network for face super-resolution[J].Journal of Computer-Aided Design & Computer Graphics，2020，32（6）：959-970.
[57] CHEN X，WANG X，LU Y，et al.RBPNET：an asymptotic residual back-projection network for super-resolution of very low-resolution face image[J].Neurocomputing，2020，376：119-127.
[58] 蒋文杰，罗晓曙，戴沁璇.基于对抗网络人脸超分辨率重建算法研究[J].计算机工程与应用，2021，57（11）：219-223.
JIANG W J，LUO X S，DAI Q X.Research on face super-resolution reconstruction algorithm based on generative adversarial networks[J].Computer Engineering and Applications，2021，57（11）：219-223.
[59] ARJOVSKY M，CHINTALA S，BOTTOU L.Wasserstein generative adversarial networks[C]//International Conference on Machine Learning，2017：214-223.
[60] 范明明，池源，张铭津，等.基于深度学习的芯片图像超分辨率重建[J].模式识别与人工智能，2019，32（4）：353-360.
FAN M M，CHI Y，ZHANG M J，et al.Chip image super-resolution reconstruction based on deep learning[J].Pattern Recognition and Artificial Intelligence，2019，32（4）：353-360.
[61] YUAN Z，HUANG H，JIANG Y，et al.An enhanced fault-detection method based on adaptive spectral decomposition and super-resolution deep learning[J].Interpretation，2019，7（3）：713-725.
[62] 施玉娟，林璐颖，杨文元.基于拉普拉斯超分辨率印刷电路板瑕疵检测方法[J].中国体视学与图像分析，2020，25（3）：295-303.
SHI Y J，LIN L Y，YANG W Y.A method for PCB defect detection based on a Laplacian super-resolution network[J].Chinese Journal of Stereology and Image Analysis，2020，25（3）：295-303.
[63] SONG H，YANG Y.Noncontact super-resolution guided wave array imaging of subwavelength defects using a multiscale deep learning approach[J].Structural Health Monitoring，2021，20（4）：1904-1923.
[64] LEE Y，YUN J W，HONG Y，et al.Accurate license plate recognition and super-resolution using a generative adversarial networks on traffic surveillance video[C]//2018 IEEE International Conference on Consumer Electronics-Asia（ICCE-Asia），2018：1-4.
[65] BíLKOVá Z，HRADI? M.Perceptual license plate super-resolution with CTC loss[J].Electronic Imaging，2020（6）.
[66] LEE S，KIM J H，HEO J P.Super-resolution of license plate images via character-based perceptual loss[C]//2020 IEEE International Conference on Big Data and Smart Computing（BigComp），2020：560-563.
[67] 王殿伟，郝元杰，刘颖，等.基于逐级反投影网络的车牌图像超分辨率重建[J].激光与光电子学进展，2020，57（16）：56-64.
WANG D W，HAO Y J，LIU Y，et al.Super-resolution reconstruction of license plate image based on gradual back-projection network[J].Laser&Optoelectronics Progress，2020，57（16）：56-64.
[68] LIU Y，DONG Z，LIM K P，et al.A densely connected face super-resolution network based on attention mechanism[C]//2020 15th IEEE Conference on Industrial Electronics and Applications（ICIEA），2020：148-152.
[69] LI J，WU L，WANG S，et al.Super resolution image reconstruction of textile based on SRGAN[C]//2019 IEEE International Conference on Smart Internet of Things（SmartIoT），2019：436-439.
[70] CHEN L，TANG R，ANISETTI M，et al.A lightweight iterative error reconstruction network for infrared image super-resolution in smart grid[J].Sustainable Cities and Society，2021，66：102520.
[71] YIN H，WAN J，ZHANG S J，et al.ADSCN：adaptive dense skip connection network for railway infrastructure displacement monitoring images super-resolution[J].Multimedia Tools and Applications，2021，80（4）：6105-6120.
[72] TODERICI G，VINCENT D，JOHNSTON N，et al.Full resolution image compression with recurrent neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：5306-5314.
[73] WANG Y，WANG L，WANG H，et al.Blind single image super-resolution with a mixture of deep networks[J].Pattern Recognition，2020，102：107169.