Review of Research on Real-World Single Image Super-Resolution Reconstruction

doi:10.3778/j.issn.1002-8331.2208-0223

Abstract

Abstract: Single image super-resolution is an important research topic in the field of computer vision in recent decades. The super-resolution reconstruction algorithm based on deep learning has made breakthroughs. When the super-resolution algorithm is applied to the image in real scene, the effect will be greatly reduced, and serious blur and ringing effect will appear. In this context, more and more researchers are committed to the study of real-world single image SR（RSISR） algorithm. Taking RSISR as the research object, this paper first introduces common image data sets and evaluation indexes, and then analyzes and compares the characteristics, performance and shortcomings of various methods from two aspects：SR methods based on external data sets and SR methods based on internal data sets. Finally, the difficulties and challenges of RSISR are discussed, and the future development trend is considered and prospected.

Key words: single image super-resolution, real scene, deep learning, super-resolution data set

摘要： 单幅图像超分辨率是近几十年来计算机视觉领域的一个重要研究课题，基于深度学习的超分辨率重建算法已经取得突破性进展，但当大多数算法应用到真实场景中的图像时效果会大大降低，出现严重模糊、振铃效应等。在此背景下越来越多研究人员致力于研究真实场景下的图像超分辨率算法（real-world single image super-resolution，RSISR）。聚焦于真实场景下图像超分辨率重建算法，介绍了常用公共图像数据集和图像评估指标，从基于外部数据集SR方法和基于内部数据集SR方法两大方向分析对比了各种方法的特点、性能和不足。讨论了RSISR存在的困难和挑战，并对未来的发展趋势进行了思考与展望。

关键词: 单幅图像超分辨率, 真实场景, 深度学习, 超分辨率数据集

ZHANG Yanqing, MA Jianhong, HAN Ying, CAO Yangjie, LI Jie, YANG Cong. Review of Research on Real-World Single Image Super-Resolution Reconstruction[J]. Computer Engineering and Applications, 2023, 59(8): 28-40.

张艳青, 马建红, 韩颖, 曹仰杰, 李颉, 杨聪. 真实场景下图像超分辨率重建研究综述[J]. 计算机工程与应用, 2023, 59(8): 28-40.

References

[1] ZHANG L，ZHANG H，SHEN H，et al.A super-resolution reconstruction algorithm for surveillance images[J].Signal Processing，2010，90（3）：848-859.
[2] GREENSPAN H.Super-resolution in medical imaging[J].The Computer Journal，2009，52（1）：43-63.
[3] 黄健，赵元元，郭苹，等.深度学习的单幅图像超分辨率重建方法综述[J].计算机工程与应用，2021，57（18）：13-23.
HUANG J，ZHAO Y Y，GUO P，et al.Survey of single image super-resolution based on deep learning[J].Computer Engineering and Applications，2021，57（18）：13-23.
[4] LEI S，SHI Z，ZOU Z.Super-resolution for remote sensing images via local-global combined network[J].IEEE Geoscience and Remote Sensing Letters，2017，14（8）：1243-1247.
[5] ZHANG S，YUAN Q，LI J，et al.Scene-adaptive remote sensing image super-resolution using a multiscale attention network[J].IEEE Transactions on Geoscience and Remote Sensing，2020，58（7）：4764-4779.
[6] DAI D，WANG Y，CHEN Y，et al.Is image super-resolution helpful for other vision tasks?[C]//2016 IEEE Winter Conference on Applications of Computer Vision（WACV），2016：1-9.
[7] GUO Z，WU G，SONG X，et al.Super-resolution integrated building semantic segmentation for multi-source remote sensing imagery[J].IEEE Access，2019，7：99381-99397.
[8] WANG L，LI D，ZHU Y，et al.Dual super-resolution learning for semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：3774-3783.
[9] PANG Y，CAO J，WANG J，et al.JCS-Net：joint classification and super-resolution network for small-scale pedestrian detection in surveillance images[J].IEEE Transactions on Information Forensics and Security，2019，14（12）：3322-3331.
[10] ZHANG Y，BAI Y，DING M，et al.KGSnet：key-point-guided super-resolution network for pedestrian detection in the wild[J].IEEE Transactions on Neural Networks and Learning Systems，2020，32（5）：2251-2265.
[11] YANG X，WU W，LIU K，et al.Long-distance object recognition with image super resolution：a comparative study[J].IEEE Access，2018，6：13429-13438.
[12] SIADARI T S，HAN M，YOON H.GSR-MAR：global super-resolution for person multi-attribute recognition[C]//ICCV Workshops，2019：1098-1103.
[13] LI X，ORCHARD M T.New edge-directed interpolation[J].IEEE Transactions on Image Processing，2001，10（10）：1521-1527.
[14] SUN J，XU Z，SHUM H Y.Image super-resolution using gradient profile prior[C]//2008 IEEE Conference on Computer Vision and Pattern Recognition，2008：1-8.
[15] FORD C，ETTER D.Wavelet basis reconstruction of nonuniformly sampled data[J].IEEE Transactions on Circuits and Systems II：Analog and Digital Signal Processing，1998，45（8）：1165-1168.
[16] BLU T，THéVENAZ P，UNSER M.Linear interpolation revitalized[J].IEEE Transactions on Image Processing，2004，13（5）：710-719.
[17] KEYS R.Cubic convolution interpolation for digital image processing[J].IEEE Transactions on Acoustics，Speech，and Signal Processing，1981，29（6）：1153-1160.
[18] SCHULTZ R R，STEVENSON R L.Extraction of high-resolution frames from video sequences[J].IEEE Transactions on Image Processing，1996，5（6）：996-1011.
[19] BANHAM M R，KATSAGGELOS A K.Digital image restoration[J].IEEE Signal Processing Magazine，1997，14（2）：24-41.
[20] NASONOV A V，KRYLOV A S.Fast super-resolution using weighted median filtering[C]//2010 20th International Conference on Pattern Recognition，2010：2230-2233.
[21] K?HLER T，B?TZ M，NADERI F，et al.Toward bridging the simulated-to-real gap：benchmarking super-resolution on real data[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2019，42（11）：2944-2959.
[22] SCHULTZ R R，STEVENSON R L.A Bayesian approach to image expansion for improved definition[J].IEEE Transactions on Image Processing，1994，3（3）：233-242.
[23] IRANI M，PELEG S.Super resolution from image sequences[C]//10th International Conference on Pattern Recognition，1990.
[24] STARK H，OSKOUI P.High-resolution image recovery from image-plane arrays，using convex projections[J].JOSA A，1989，6（11）：1715-1726.
[25] MEI Y，FAN Y，ZHOU Y.Image super-resolution with non-local sparse attention[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：3517-3526.
[26] GUO Y，CHEN J，WANG J，et al.Closed-loop matters：dual regression networks for single image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：5407-5416.
[27] ZHANG Y，TIAN Y，KONG Y，et al.Residual dense network for image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：2472-2481.
[28] LIM B，SON S，KIM H，et al.Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops，2017：136-144.
[29] KIM J，LEE J K，LEE K M.Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：1646-1654.
[30] WANG Y，PERAZZI F，MCWILLIAMS B，et al.A fully progressive approach to single-image super-resolution[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops（CVPRW），2018.
[31] CHEN H，HE X，QING L，et al.Real-world single image super-resolution：a brief review[J].Information Fusion，2022，79：124-145.
[32] ZHANG X，DONG H，HU Z，et al.Gated fusion network for degraded image super resolution[J].International Journal of Computer Vision，2020，128（6）：1699-1721.
[33] DAI T，CAI J，ZHANG Y，et al.Second-order attention network for single image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：11065-11074.
[34] HARIS M，SHAKHNAROVICH G，UKITA N.Deep back-projection networks for super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：1664-1673.
[35] ZHANG K，WANG B，ZUO W，et al.Joint learning of multiple regressors for single image super-resolution[J].IEEE Signal Processing Letters，2015，23（1）：102-106.
[36] TIMOFTE R，DE SMET V，VAN GOOL L.Anchored neighborhood regression for fast example-based super-resolution[C]//Proceedings of the IEEE International Conference on Computer Vision，2013：1920-1927.
[37] ZHANG Y，LI K，LI K，et al.Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：286-301.
[38] LIU J，ZHANG W，TANG Y，et al.Residual feature aggregation network for image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：2359-2368.
[39] BELL-KLIGLER S，SHOCHER A，IRANI M.Blind super-resolution kernel estimation using an internal-GAN[C]//Advances in Neural Information Processing Systems，2019.
[40] CAI J，ZENG H，YONG H，et al.Toward real-world single image super-resolution：a new benchmark and a new model[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：3086-3095.
[41] WEI P，XIE Z，LU H，et al.Component divide-and-conquer for real-world image super-resolution[C]//European Conference on Computer Vision.Cham：Springer，2020：101-117.
[42] ZHANG X，CHEN Q，NG R，et al.Zoom to learn，learn to zoom[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：3762-3770.
[43] CHEN C，XIONG Z，TIAN X，et al.Camera lens super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：1652-1660.
[44] IGNATOV A，KOBYSHEV N，TIMOFTE R，et al.Dslr-quality photos on mobile devices with deep convolutional networks[C]//ICCV，2017：3297-3305.
[45] LI X，CAO G，ZHANG Y，et al.Combining synthesis sparse with analysis sparse for single image super-resolution[J].Signal Processing：Image Communication，2020，83：115805.
[46] SHEIKH H R，SABIR M F，BOVIK A C.A statistical evaluation of recent full reference image quality assessment algorithms[J].IEEE Transactions on image processing，2006，15（11）：3440-3451.
[47] WANG Z，BOVIK A C，SHEIKH H R，et al.Image quality assessment：from error visibility to structural similarity[J].IEEE Transactions on Image Processing，2004，13（4）：600-612.
[48] SHEIKH H R，BOVIK A C，DE VECIANA G.An information fidelity criterion for image quality assessment using natural scene statistics[J].IEEE Transactions on Image Processing，2005，14（12）：2117-2128.
[49] ZHANG R，ISOLA P，EFROS A A，et al.The unreasonable effectiveness of deep features as a perceptual metric[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：586-595.
[50] EFRAT N，GLASNER D，APARTSIN A，et al.Accurate blur models vs. image priors in single image super-resolution[C]//Proceedings of the IEEE International Conference on Computer Vision，2013：2832-2839.
[51] SHAO W Z，ELAD M.Simple，accurate，and robust nonparametric blind super-resolution[M]//Image and graphics.Berlin：Springer，2015：333-348.
[52] GU J，LU H，ZUO W，et al.Blind super-resolution with iterative kernel correction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：1604-1613.
[53] CORNILLERE V，DJELOUAH A，YIFAN W，et al.Blind image super-resolution with spatially variant degradations[J].ACM Transactions on Graphics（TOG），2019，38（6）：1-13.
[54] HUANG Y，LI S，WANG L，et al.Unfolding the alternating optimization for blind super resolution[C]//Advances in Neural Information Processing Systems，2020：5632-5643.
[55] KIM S Y，SIM H，KIM M.Koalanet：blind super-resolution using kernel-oriented adaptive local adjustment[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：10611-10620.
[56] ZHOU R，SUSSTRUNK S.Kernel modeling super-resolution on real low-resolution images[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：2433-2443.
[57] JI X，CAO Y，TAI Y，et al.Real-world super-resolution via kernel estimation and noise injection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，2020：466-467.
[58] WANG X，XIE L，DONG C，et al.Real-esrgan：training real-world blind super-resolution with pure synthetic data[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：1905-1914.
[59] LIANG J，SUN G，ZHANG K，et al.Mutual affine network for spatially variant kernel estimation in blind image super-resolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：4096-4105.
[60] ZHANG K，LIANG J，VAN GOOL L，et al.Designing a practical degradation model for deep blind image super-resolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：4791-4800.
[61] AAKERBERG A，NASROLLAHI K，MOESLUND T B.Real‐world super‐resolution of face‐images from surveillance cameras[J].IET Image Processing，2022，16（2）：442-452.
[62] WANG L，WANG Y，DONG X，et al.Unsupervised degradation representation learning for blind super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：10581-10590.
[63] BULAT A，YANG J，TZIMIROPOULOS G.To learn image super-resolution，use a GAN to learn how to do image degradation first[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：185-200.
[64] ZHU J Y，PARK T，ISOLA P，et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2223-2232.
[65] PRAJAPATI K，CHUDASAMA V，PATEL H，et al.Unsupervised single image super-resolution network（USISResNet） for real-world data using generative adversarial network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，2020：464-465.
[66] YOU C，LI G，ZHANG Y，et al.CT super-resolution GAN constrained by the identical，residual，and cycle learning ensemble（GAN-CIRCLE）[J].IEEE Transactions on Medical Imaging，2019，39（1）：188-203.
[67] MAEDA S.Unpaired image super-resolution using pseudo-supervision[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：291-300.
[68] LIAN S，ZHOU H，SUN Y.FG-SRGAN：a feature-guided super-resolution generative adversarial network for unpaired image super-resolution[C]//International Symposium on Neural Networks.Cham：Springer，2019：151-161.
[69] YOON K.Simple and efficient unpaired real-world super-resolution using image statistics[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：1983-1990.
[70] YUAN Y，LIU S，ZHANG J，et al.Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops，2018：701-710.
[71] KIM G，PARK J，LEE K，et al.Unsupervised real-world super resolution with cycle generative adversarial network and domain discriminator[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，2020：456-457.
[72] WANG W，ZHANG H，YUAN Z，et al.Unsupervised real-world super-resolution：a domain adaptation perspective[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：4318-4327.
[73] MOSSERI I，ZONTAK M，IRANI M.Combining the power of internal and external denoising[C]//IEEE International Conference on Computational Photography（ICCP），2013：1-9.
[74] ZONTAK M，IRANI M.Internal statistics of a single natural image[C]//CVPR 2011，2011：977-984.
[75] GLASNER D，BAGON S，IRANI M.Super-resolution from a single image[C]//IEEE International Conference on Computer Vision，2009.
[76] MICHAELI T，IRANI M.Nonparametric blind super-resolution[C]//IEEE International Conference on Computer Vision，2013.
[77] SHOCHER A，COHEN N，IRANI M.“Zero-shot” super-resolution using deep internal learning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：3118-3126.
[78] YAMAC M，ATAMAN B，NAWAZ A.Kernelnet：a blind super-resolution kernel estimation network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：453-462.
[79] EMAD M，PEEMEN M，CORPORAAL H.Dualsr：zero-shot dual learning for real-world super-resolution[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision，2021：1630-1639.
[80] SOH J W，CHO S，CHO N I.Meta-transfer learning for zero-shot super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：3516-3525.
[81] PARK S，YOO J，CHO D，et al.Fast adaptation to super-resolution networks via meta-learning[C]//European Conference on Computer Vision.Cham：Springer，2020：754-769.
[82] LI S，ZHANG G，LUO Z，et al.From general to specific：online updating for blind super-resolution[J].Pattern Recognition，2022，127：108613.