Image Super-Resolution Based on Blur Kernel Correction with Unknown Degradation Method

doi:10.3778/j.issn.1002-8331.2103-0530

Abstract

Abstract: Real-world images contain unknown blur and noise information, and the existing datasets using bicubic downsampling as the degradation method cannot train super-resolution networks suitable for real-world images. Therefore, a predictor is designed to estimate the blur kernel and a corrector is designed to obtain precise image blur kernel information. High-resolution images are processed by blurring and injecting noise to construct low-resolution real-world images, which are utilized to train a new super-resolution network. After each convolutional layer, spatial feature transformation is performed on the feature map according to the blur kernel, which improved the network’s ability to process different blurred images. The non-linear mapping part is connected with a residual dense block structure and integrated into the generative adversarial network framework to enhance the ability of texture detail recovery. The experiment results on the datasets of Flickr2K and DIV2K show that the peak signal-to-noise ratio, structural similarity and perception index of the new method are higher than those of classic methods such as EDSR and ESRGAN.

Key words: real-world image, , kernel, , residual dense block, , spatial feature transformation, , perception index

摘要： 真实世界图像包含未知模糊和噪声信息，现有以双三次下采样为退化方式的数据集无法训练出适用于真实世界图像的超分辨率网络。为此设计预测器初步估计模糊核并设计校正器得到准确的图像模糊核信息。高分辨率图像通过模糊化处理并注入噪声构建低分辨率真实世界图像。设计新型超分辨率网络结构，每个卷积层后根据模糊核对特征图进行空间特征变换，提高网络处理不同模糊图像的能力。将非线性映射部分以残差密集块结构相连，并整合入生成对抗网络框架增强纹理细节恢复能力。在Flickr2K和DIV2K两个数据集上的测试结果表明新方法的峰值信噪比、结构相似度和感知指数高于EDSR和ESRGAN等经典方法。

关键词: 真实世界图像, 模糊核, 残差密集块, 空间特征变换, 感知指数

WANG Lan, KONG Xiangyi, ZHANG Haitao. Image Super-Resolution Based on Blur Kernel Correction with Unknown Degradation Method[J]. Computer Engineering and Applications, 2022, 58(21): 232-242.

汪澜, 孔祥屹, 张海涛. 基于模糊核校正的未知退化方式图像超分辨率[J]. 计算机工程与应用, 2022, 58(21): 232-242.

References

[1] 曾凯，丁世飞.图像超分辨率重建的研究进展[J].计算机工程与应用，2017，53（16）：29-35.
ZENG K，DING S F.Advances in image super-resolution reconstruction[J].Computer Engineering and Applications，2017，53（16）：29-35.
[2] DONG C，LOY C C，HE K，et al.Image super-resolution using deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2015，38（2）：295-307.
[3] DONG C，LOY C C，TANG X.Accelerating the super-resolution convolutional neural network[C]//European Conference on Computer Vision.Cham：Springer，2016：391-407.
[4] SHI W，CABALLERO J，HUSZáR F，et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：1874-1883.
[5] KIM J，LEE J K，LEE K M.Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：1646-1654.
[6] GHIFARY M，KLEIJN W B，ZHANG M，et al.Deep reconstruction-classification networks for unsupervised domain adaptation[C]//European Conference on Computer Vision.Cham：Springer，2016：597-613.
[7] MAO X J，SHEN C，YANG Y B.Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections[J].arXiv：1603.09056，2016.
[8] TAI Y，YANG J，LIU X.Image super-resolution via deep recursive residual network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：3147-3155.
[9] LIM B，SON S，KIM H，et al.Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops，2017：136-144.
[10] LEDIG C，THEIS L，HUSZáR F，et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：4681-4690.
[11] 李现国，冯欣欣，李建雄.多尺度残差网络的单幅图像超分辨率重建[J].计算机工程与应用，2021，57（7）：215-221.
LI X G，FENG X X，LI J X.Sigle image super-resolution reconstruction based on multi-scale residual network[J].Computer Engineering and Applications，2021，57（7）：215-221.
[12] 郭继昌，吴洁，郭春乐，等.基于残差连接卷积神经网络的图像超分辨率重构[J].吉林大学学报（工学版），2019，49（5）：1726-1734.
GUO J C，WU J，GUO C L，et al.Image super-resolution reconstruction based on residual connection convolutional neural network[J].Journal of Jilin University（Engineering and Technology Edition），2019，49（5）：1726-1734.
[13] TONG T，LI G，LIU X，et al.Image super-resolution using dense skip connections[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：4799-4807.
[14] WANG X，YU K，WU S，et al.Esrgan：enhanced super-resolution generative adversarial networks[C]//Proceedings of the European Conference on Computer Vision（ECCV） Workshops，2018.
[15] BELL-KLIGLER S，SHOCHER A，IRANI M.Blind super-resolution kernel estimation using an internalgan[J].arXiv：1909.06581，2019.
[16] SHOCHER A，COHEN N，IRANI M.“zero-shot” super-resolution using deep internal learning[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：3118-3126.
[17] ZHANG K，ZUO W，ZHANG L.Deep plug-and-play super-resolution for arbitrary blur kernels[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：1671-1681.
[18] ZHOU R，SUSSTRUNK S.Kernel modeling super-resolution on real low-resolution images[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：2433-2443.
[19] GULRAJANI I，AHMED F，ARJOVSKY M，et al.Improved training of wasserstein gans[J].arXiv：1704. 00028，2017.
[20] GU J，LU H，ZUO W，et al.Blind super-resolution with iterative kernel correction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：1604-1613.
[21] WANG X，YU K，DONG C，et al.Recovering realistic texture in image super-resolution by deep spatial feature transform[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：606-615.
[22] IOFFE S，SZEGEDY C.Batch normalization：accelerating deep network training by reducing internal covariate shift[C]//International Conference on Machine Learning，2015：448-456.
[23] HUANG G，LIU Z，VAN DER MAATEN L，et al.Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：4700-4708.
[24] ZHANG K，ZUO W，ZHANG L.Learning a single convolutional super-resolution network for multiple degradations[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：3262-3271.
[25] ZHOU R，SUSSTRUNK S.Kernel modeling super-resolution on real low-resolution images[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：2433-2443.
[26] IGNATOV A，KOBYSHEV N，TIMOFTE R，et al.Dslr-quality photos on mobile devices with deep convolutional networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：3277-3285.
[27] TIMOFTE R，AGUSTSSON E，VAN GOOL L，et al.Ntire 2017 challenge on single image super-resolution：methods and results[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops，2017：114-125.
[28] AGUSTSSON E，TIMOFTE R.Ntire 2017 challenge on single image super-resolution：dataset and study[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops，2017：126-135.
[29] KRIZHEVSKY A，SUTSKEVER I，HINTON G E.Imagenet classification with deep convolutional neural networks[J].Advances in Neural Information Processing Systems，2012，25：1097-1105.
[30] BEVILACQUA M，ＲOUMY A，GUILLEMOT C，et al.Low-complexity single-image super-resolution based on nonnegative neighbor embedding[C]//Proceedings of 2012 British Machine Vision Conference，2012.
[31] ZEYDE Ｒ，ELAD M，PROTTER M.On single image scale-up using sparse-representations[C]//Proceedings of the 7th International Conference.Berlin，Heidelberg：Springer-Verlag，2012：711-730．
[32] MARTIN D，FOWLKES C，TAL D，et al.A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings of the 8th IEEE International Conference on Computer Vision.Vancouver，BC，Canada：IEEE，2002：416-423．