Real-World Image Super-Resolutioin Based on Noise and U-Shape Discrimination Network

doi:10.3778/j.issn.1002-8331.2111-0393

Abstract

Abstract: Previous image super-resolution algorithms based on convolutional neural networks are usually trained on synthetic ideal datasets, and their performance will drop dramatically in real-world scenarios. To better extract the original feature information in real-world images and model their degradation process, this paper proposes a real-world image super-resolution reconstruction based on noise and U-shape discrimination network. To make the degraded image having a similar feature distribution to the source image and restoring more detailed information and better visual quality, this paper uses the complex noise information directly collected from the source real-world image to inject the synthetic degraded image. In addition, this paper proposes to employ a U-shape discrimination network with spectral normalization to increase the discrimination network capability and stabilize the training, and suppress the appearance of artifacts in image reconstruction. Experimental results on three benchmark datasets show that compared with the state-of-the-art methods, this model achieves the best results in all three evaluation protocols （peak signal-to-noise ratio（PSNR）, structural similarity（SSIM）, and learned perceptual image patch similarity（LPIPS）） and has better visual quality.

Key words: super-resolution, real-world image, noise, degradation processing, U-shape discrimination network

摘要： 以往基于卷积神经网络的图像超分辨率算法往往是在理想的合成数据进行训练，当应用在真实世界（Real-World）场景时性能就会严重下降。为更好地提取出Real-World图像中的原有特征信息，为其降质过程建模，提出一种噪声和U型判别网络的Real-World图像超分辨率算法。利用直接收集到的Real-World图像原有的复杂噪声信息，结合合成的降质图像，达到降质后图像与源图像保持特征分布相似的目的，以恢复更多的细节信息和更好的观感。此外，提出使用频谱归一正则化的U型判别网络，以提高判别网络的能力和稳定训练，抑制图像重建中伪影的出现。在三个基准数据集上的实验结果表明，与最新的方法相比，该模型在三个评价指标（峰值信噪比、结构相似度和感知图像块相似度）上均取得了最好的结果，且有着更好的观感效果。

关键词: 超分辨率, 真实世界图像, 噪声, 降质过程, U型判别网络

LI Hao, YANG Zhijing, WANG Meilin, LING Wing-Kuen. Real-World Image Super-Resolutioin Based on Noise and U-Shape Discrimination Network[J]. Computer Engineering and Applications, 2023, 59(6): 204-211.

李灏, 杨志景, 王美林, 凌永权. 噪声和U型判别网络的真实世界图像超分辨率[J]. 计算机工程与应用, 2023, 59(6): 204-211.

References

[1] DONG C，LOY C C，HE K，et al.Image super-resolution using deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2016，38（2）：295-307.
[2] 李诚，张羽，黄初华.改进的生成对抗网络图像超分辨率重建[J].计算机工程与应用，2020，56（4）：191-196.
LI C，ZHANG Y，HUANG C H.Improved super-resolution reconstruction of image based on generative adversarial networks[J].Computer Engineering and Applications，2020，56（4）：191-196.
[3] 刘璟，宋海川，黄建设，等.通道与空间注意力图像超分辨率网络[J].计算机工程与应用，2021，57（2）：209-216.
LIU J，SONG H C，HUANG J S，et al.Channel and spatial attention image super resolution network[J].Computer Engineering and Applications，2021，57（2）：209-216.
[4] LIM B，SON S，KIM H，et al.Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.Washington D C，USA：IEEE Press，2017：71-98.
[5] ZHANG Y，LI K P，LI K，et al.Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the European Conference on Computer Vision.Berlin，Germany：Springer，2018：286-301.
[6] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial nets[EB/OL].[2019-11-25].https：//www.researchgate.net/publication/319770355_Generative_
Adversarial_Nets.
[7] LEDIG C，THEIS L，HUSZAR F，et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Washington D C，USA：IEEE Press，2017：4681-4690.
[8] LUGMAYR A，DANELLJAN M，TIMOFTE R，et al.NTIRE 2020 challenge on real-world image super-resolution：methods and results[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.Washington D C，USA：IEEE Press，2020：2058-2076.
[9] LUGMAYR A，DANELLJAN M，TIMOFTE R，et al.AIM 2019 challenge on real-world image super-resolution：methods and results[C]//Proceedings of the IEEE International Conference on Computer and Vision Workshops.Washington D C，USA：IEEE Press，2019：3575-3583.
[10] JI X，CAO Y，TAI Y，et al.Real-world super-resolution via kernel estimation and noise injection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.Washington D C，USA：IEEE Press，2020：1914-1923.
[11] FRITSCHE M，GU S，TIMOFTE R.Frequency separation for real-world super-resolution[C]//Proceedings of the IEEE International Conference on Computer and Vision Workshops.Washington D C，USA：IEEE Press，2019：3599-3608.
[12] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409.
1556，2014.
[13] WANG X，YU K，WU S，et al.ESRGAN：enhanced super-resolution generative adversarial networks[C]//Proceedings of the European Conference on Computer Vision.Berlin，Germany：Springer，2018：63-79.
[14] SHOCHER A，COHEN N，IRANI M.“Zero-Shot” super-resolution using deep internal learning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Washington D C，USA：IEEE Press，2018：3118-3126.
[15] WEI Y，GU S，LI Y，et al.Unsupervised real-world image super resolution via domain-distance aware training[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Washington D C，USA：IEEE Press，2021：13385-13394.
[16] YUAN Y，LIU S，ZHANG J，et al.Unsupervised image super-resolution using cycle-in-cycle generative adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.Washington D C，USA：IEEE Press，2018：701-710.
[17] SHI W，CABALLERO J，HUSZAR F，et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.Washington D C，USA：IEEE Press，2016：1874-1883.
[18] CAI J，ZENG H，YONG H，et al.Toward real-world single image super-resolution：a new benchmark and a new model[C]//Proceedings of the IEEE International Conference on Computer and Vision Workshops.Washington D C，USA：IEEE Press，2019：3086-3095.
[19] ZHANG R，ISOLA P，EFROS A A，et al.The unreasonable effectiveness of deep features as a perceptual metric[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops.Washington D C，USA：IEEE Press，2018：586-595.
[20] KINGMA D P，BA J.Adam：a method for stochastic optimization[J].arXiv：1412.6980v6，2015.