Super-Resolution Reconstruction of Single-Frame Images in Cross-Scale Multi-Branching Networks

doi:10.3778/j.issn.1002-8331.2203-0307

Abstract

Abstract: Super-resolution reconstruction algorithm is one of the key research problems in the field of computer vision. At present, the image requirements in various fields are gradually increasing, and the existing super-resolution reconstruction algorithms improve the image quality by deepening a single network depth, ignoring the computational complexity problem when reconstructing, and lacking the use of image own information to improve the image reconstruction effect. To address the above problems, a cross-scale multi-branching single-frame image supe-resolution reconstruction network is proposed. The cross-scale module explores the correlation of similar regions within the image and is used to enhance the ability to reconstruct image detail information and obtain better visual effects. The multi-branching structure reconstructs regions in the image with different reconstruction difficulty by networks of different complexity separately, solving the problem of high computational complexity of deep networks. Experimental results show that the model has higher peak signal-to-noise ratio and structural similarity on Urban100 and Manga109 test sets compared with other methods.

Key words: super-resolution reconstruction, convolutional neural network, cross-scale, multi-branching

摘要： 超分辨率重建算法是计算机视觉领域的重点研究问题之一，目前各个领域对图像要求逐渐提高，现有的超分辨率重建算法通过加深单一网络深度来提升图像质量，忽略了重建时的计算复杂度问题，同时缺少利用图像自身信息来提升图像的重建效果。针对以上问题提出了一种跨尺度多分支的单帧图像超分辨率重建网络，跨尺度模块探索图像内部相似区域的相关性，用于提升重建图像细节信息的能力，并获得更好的视觉效果。多分支结构将图像中重建难度不同的区域，通过不同复杂度的网络分别进行重建，解决了深度网络计算复杂度高的问题。实验结果表明，该模型在Urban100和Manga109测试集上相比其他方法具有更高的峰值信噪比和结构相似度。

关键词: 超分辨率重建, 卷积神经网络, 跨尺度, 多分支

LIU Nanyan, MA Shengxiang, Wei Hongfei. Super-Resolution Reconstruction of Single-Frame Images in Cross-Scale Multi-Branching Networks[J]. Computer Engineering and Applications, 2022, 58(19): 257-266.

刘南艳, 马圣祥, 魏鸿飞. 跨尺度多分支网络的单帧图像超分辨率重建[J]. 计算机工程与应用, 2022, 58(19): 257-266.

References

[1] 席志红，侯彩燕，袁昆鹏.基于残差网络的医学图像超分辨率重建[J].计算机工程与应用，2019，55（19）：191-197.
XI Z H，HOU C Y，YUAN K P.Medical image super resolution reconstruction based on residual network[J].Computer Engineering and Applications，2019，55（19）：191-197.
[2] LI Y，SIXOU B，PEYRIN F.A review of the deep learning methods for medical images super resolution problems[J].IRBM，2021，42（2）：120-133.
[3] QIU D，ZHENG L，ZHU J，et al.Multiple improved residual networks for medical image super-resolution[J].Future Generation Computer Systems，2021，116：200-208.
[4] 邢晓羊，魏敏，符颖.基于特征损失的医学图像超分辨率重建[J].计算机工程与应用，2018，54（20）：202-207.
XING X Y，WEI M，FU Y.Super-resolution reconstruction of medical images using feature-based loss[J].Computer Engineering and Applications，2018，54（20）：202-207.
[5] 蒋文杰，罗晓曙，戴沁璇.基于对抗网络人脸超分辨率重建算法研究[J].计算机工程与应用，2021，57（11）：219-223.
JIANG W J，LUO X S，DAI Q X.Research on face super-resolution reconstruction algorithm based on generative adversarial networks[J].Computer Engineering and Applications，2021，57（11）：219-223.
[6] CHEN Y，PHONEVILAY V，TAO J，et al.The face image super-resolution algorithm based on combined representation learning[J].Multimedia Tools and Applications，2021，80：30839-30861.
[7] LI M，ZHANG Z，YU J，et al.Learning face image super-resolution through facial semantic attribute transformation and self-attentive structure enhancement[J].IEEE Transactions on Multimedia，2020，23：468-483.
[8] 刘颖，董占龙，卢津，等.多注意力域的稠密连接人脸超分算法[J].计算机工程与应用，2020，56（20）：179-185.
LIU Y，DONG Z L，LU J，et al.Densely connected face super-division algorithm with multiple attention domains[J].Computer Engineering and Applications，2020，56（20）：179-185.
[9] 夏皓，吕宏峰，罗军，等.图像超分辨率深度学习研究及应用进展[J].计算机工程与应用，2021，57（24）：51-60.
XIA H，LYU H F，LUO J，et al.Survey on deep learning based image super-resolution[J].Computer Engineering and Applications，2021，57（24）：51-60.
[10] LU T，WANG J，ZHANG Y，et al.Satellite image super-resolution via multi-scale residual deep neural network[J].Remote Sensing，2019，11（13）：1588.
[11] SHIN C，KIM S，KIM Y.Satellite image target super-resolution with adversarial shape discriminator[J].IEEE Geoscience and Remote Sensing Letters，2020，99：1-5.
[12] 贺智，贺丹.基于深度学习的高分四号卫星图像超分辨率重建[J].遥感学报，2020，24（12）：1500-1510.
HE Z，HE D.Deep learning-based super-resolution for GF?4 satellite imagery[J].Journal of Remote Sensing，2020，24（12）：1500-1510.
[13] LAN R，SUN L，LIU Z，et al.Cascading and enhanced residual networks for accurate single-image super-resolution[J].IEEE Transactions on Cybernetics，2020，51（1）：115-125.
[14] KONG X，ZHAO H，QIAO Y，et al.Classsr：A general framework to accelerate super-resolution networks by data characteristic[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：12016-12025.
[15] HU Y，GAO X，LI J，et al.Single image super-resolution with multi-scale information cross-fusion network[J].Signal Processing，2021，179：107831.
[16] 李现国，冯欣欣，李建雄.多尺度残差网络的单幅图像超分辨率重建[J].计算机工程与应用，2021，57（7）：215-221.
LI X G，FENG X X，LI J X.Sigle image super-resolution reconstruction based on multi-scale residual network[J].Computer Engineering and Applications，2021，57（7）：215-221.
[17] 刘颖，朱丽，林庆帆，等.图像超分辨率技术的回顾与展望[J].计算机科学与探索，2020，14（2）：181-199.
LIU Y，ZHU L，LIM K P，et al.Review and prospect of image super-resolution technology[J].Journal of Frontiers of Computer Science and Technology，2020，14（2）：181-199.
[18] 唐家军，刘辉，胡雪影.功能型复合深度网络的图像超分辨率重建[J].计算机科学与探索，2020，14（8）：1368-1379.
TANG J J，LIU H，HU X Y.Image super-resolution reconstruction of functional composite deep network[J].Journal of Frontiers of Computer Science and Technology，2020，14（8）：1368-1379.
[19] ISOBE T，JIA X，GU S，et al.Video super-resolution with recurrent structure-detail network[C]//Proceedings of European Conference on Computer Vision，2020.
[20] LIU C，SUN D.A Bayesian approach to adaptive video super resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2011：209-216.
[21] TIAN Y，ZHANG Y，FU Y，et al.TDAN：Temporally-deformable alignment network for video super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：3360-3369.
[22] HABIB Z.DSTNet：Deformable spatio-temporal convolutional residual network for video super-resolution[J].Mathematics，2021，9.
[23] TSAI R Y，HUANG T S.Multipleframe image restoration and registration[C]//Advances in Computer Vision and Image Processing，1984：317-339.
[24] CHAO D，CHEN C L，HE K，et al.Learning a deep convolutional network for image super-resolution[C]//Proceedings of European Conference on Computer Vision，2014.
[25] CHAO D，CHEN C L，TANG X.Accelerating the super-resolution convolutional neural network[C]//Proceedings of European Conference on Computer Vision，2016.
[26] KIM J，LEE J K，LEE K M.Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of IEEE Conference on Computer Vision & Pattern Recognition，2016.
[27] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of IEEE Conference on Computer Vision & Pattern Recognition，2016.
[28] LIM B，SON S，KIM H，et al.Enhanced deep residual networks for single image super-resolution[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops，2017.
[29] 汪鑫耘，李丹.生成式对抗网络在超分辨率图像重建中的应用[J].计算机科学与探索，2020，14（4）：680-687.
WANG X Y，LI D.Application of generative adversarial network in super-resolution image reconstruction[J].Journal of Frontiers of Computer Science and Technology，2020，14（4）：680-687.
[30] 彭晏飞，高艺，杜婷婷，等.生成对抗网络的单图像超分辨率重建方法[J].计算机科学与探索，2020，14（9）：1612-1620.
PENG Y F，GAO Y，DU T T，et al.Single image super-resolution reconstruction method for generative adversarial network[J].Journal of Frontiers of Computer Science and Technology，2020，14（9）：1612-1620.
[31] LEDIG C，THEIS L，F HUSZAR，et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017.
[32] WANG X，YU K，WU S，et al.ESRGAN：Enhanced super-resolution generative adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops，2018.
[33] AHN N，KANG B，SOHN K A.Fast，accurate，and lightweight super-resolution with cascading residual network[C]//Proceedings of the European Conference on Computer Vision，2018：252-268.
[34] AGUSTSSON E，TIMOFTE R.Ntire 2017 challenge on single image super-resolution：Dataset and study[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops，2017：126-135.