Research on Multi-Feature Fusion Image Restoration Based on Edge Conditions

doi:10.3778/j.issn.1002-8331.2206-0356

Abstract

Abstract: Aiming at the lack of rational reasoning of the deep structure inside the image loss area and how to generate more accurate and detailed texture information of images in the current image inpainting field,a multi-feature fusion image inpainting method based on edge conditions is proposed, which named MEGAN（multi-feature fusion network model based on edge condition）. The model adopts a two-stage generation idea. Firstly, the edge generation adversarial network is used to repair the edge information of the defective image. Secondly, the complete edge information is used to help the texture detail network to generate a complete image. A gated convolution is added to the generator structure to reduce the interference of invalid pixels on the repair process, and a gated multi-extension convolution block（GM block） is introduced to achieve multi-scale feature extraction of the image to be repaired.The multi-scale spectrally normalized Markov discriminator not only promotes structural coherence and detailed representation of generated images, but also strictly controls the magnitude of gradient variation, thereby improving model accuracy and stabilizing training. The test results on celebA and Places2 datasets show that MEGAN significantly outperforms mainstream image inpainting algorithms in generating reasonable image structure and clear detailed texture.

Key words: deep learning, image restoration, adversarial network generation, gated convolution, multi-feature fusion

摘要： 针对当前图像修复领域存在的缺乏对图像损失区域深层结构的合理性推理问题，以及如何生成更加准确清晰的纹理信息提出一种基于边缘条件的多特征融合图像修复方法——MEGAN（multi-feature fusion network model based on edge condition）。模型采用两阶段生成思想，使用边缘生成对抗网络修复缺损图像的边缘信息；用完整的边缘信息帮助纹理细节网络生成完整图像。在生成器结构上添加门控卷积以减少无效像素对修复过程的干扰，带门控的多扩张卷积块（gated multi-extension convolution block，GM block）实现对待修复图像的多尺度特征提取。多尺度谱归一化马尔可夫判别器在促进生成图像的结构一致性和细节表现力的同时严格控制梯度变化幅度，从而提高模型精度，稳定训练。在celebA和Places2数据集上的测试结果显示，MEGAN在生成合理的图像结构和准确清晰的细节纹理上明显优于主流的图像修复算法。

关键词: 深度学习, 图像修复, 生成对抗网络, 门控卷积, 多特征融合

OU Jing, WEN Zhicheng, DENG Wengui, ZHANG Shuting. Research on Multi-Feature Fusion Image Restoration Based on Edge Conditions[J]. Computer Engineering and Applications, 2023, 59(23): 191-201.

欧静, 文志诚, 邓文贵, 张姝婷. 利用边缘条件的多特征融合图像修复算法[J]. 计算机工程与应用, 2023, 59(23): 191-201.

References

[1] QIN Z，ZENG Q L，ZONG Y X，et al.Image inpainting based on deep learning：a review[J].Displays，2021，69：102028.
[2] ROJAS D J B，FERNANDES B J T，FERNANDES S M M.A review on image inpainting techniques and datasets[C]//2020 33rd Conference on Graphics，Patterns and Images（SIBGRAPI）.Piscataway：IEEE，2020：240-247.
[3] 姜艺，胥加洁，柳絮，等.边缘指导图像修复算研究[J].计算机科学与探索，2022，16（3）：669-682.
JIANG Y，XV J J，LIU X，et al.Research on edge-guided image repair algorithm[J].Journal of Frontiers of Computer Science and Technology，2022，16（3）：669-682.
[4] WAN Z Y，ZHANG B，CHEN D D，et al.Bringing old photos back to life[J].arXiv：2004.09484，2020.
[5] HASSAN M，WANG Y，WANG D，et al.Restorable-inpainting：a novel deep learning approach for shoeprint restoration[J].Information Sciences，2022，600：22-42
[6] ROY H，CHAUDHURY S，YAMASAKI T，et al.Toward better planetary surface exploration by orbital imagery inpainting[J].IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing，2021，14：175-189.
[7] LI C，DING Y，YU B，et al.Inpainting of continuous frames of old movies based on deep neural network[C]//2018 International Conference on Audio，Language and Image Processing（ICALIP），Shanghai，China，Jul 16-17，2018.Piscataway：IEEE，2018：132-137.
[8] KORNILOV A，SAFONOV I，REIMERS I，et al.Deep neural networks for ring artifacts segmentation and corrections in fragments of CT images[C]//2021 28th Conference of Open Innovations Association（FRUCT），Moscow，Russia，Jan 27-29，2021.Piscataway：IEEE，2021：181-193.
[9] NAZERI K，NG E，JOSEPH T，et al.EdgeConnect：stru-cture guided image inpainting using edge prediction[C]//2019 IEEE/CVF International Conference on Computer Vision Workshop（ICCVW），Seoul，Korea （South），Oct 27-28，2019.Piscataway：IEEE，2019：3265-3274.
[10] YU J，LIN Z，YANG J，et al.Free-form image inpainting with gated convolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision（ICCV），2019：4471-4480.
[11] 高杰，霍智勇.一种门控卷积生成对抗网络的图像修复算法[J].西安电子科技大学学报，2022，49（1）：216-224.
GAO J，HUO Z Y.Algorithm for image inpainting in generative adversarial networks based on gated convolution[J].Journal of Xidian University，2022，49（1）：216-224.
[12] CHEN L C，PAPANDREOU G，KOKKINOS I，et al.DeepLab：semantic image segmentation with deep convolutional nets，atrous convolution，and fully connected CRFs[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2018，40：834-848.
[13] CHEN L C，PAPANDREOU G，SCHROFF F，et al.Rethinking atrous convolution for semantic image segmentation[J].arXiv：1706.05587，2017.
[14] MIYATO T，KATAOKA T，KOYAMA M，et al.Spectral normalization for generative adversarial networks[J].arXiv：1802.05957，2018.
[15] ISOLA P，ZHU J Y，ZHOU T，et al.Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and pattern recognition（CVPR），Honolulu，HI，USA，July 21-26，2017.Piscataway：IEEE，2017：1125-1134.
[16] WANG T C，LIU M Y，ZHU J Y，et al.High-resolution image synthesis and semantic manipulation with conditional gans[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition（CVPR），Salt Lake City，UT，USA，June 18-22，2018.Piscataway：IEEE，2018：8798-8807.
[17] BERTALMIO M，SAPIRO G，CASELLES V，et al.Image inpainting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques.New York：ACM Press/Addison-Wesley Publishing Co，2000：417-424.
[18] SHEN J，CHAN T F.Mathematical models for local nontexture inpaintings[J].SIAM Journal on Applied Mathematics，2002，62（3）：1019-1043.
[19] CHAN T F，SHEN J H.Mathematical models for local nontexture inpaintings[J].Journal of Visual Communication and Image Representation，2001，12（4）：436-449.
[20] CRIMINISI A，PEREZ P，TOYAMA K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing，2004，13（9）：1200-1212.
[21] DARABI S，SHECHTMAN E，BARNES C，et al.Image melding：combining inconsistent images using patch-based synthesis[J].ACM Transactions on Graphics（ToG），2012，31（4）：1-10.
[22] HAYS J，EFROS A A.Scene completion using millions of photographs[J].ACM Transactions on Graphics（ToG），2007，26（3）：87-94.
[23] PATHAK D，KRAHENBUHL P，DONAHUE J，et al.Context encoders：feature learning by inpainting[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition（CVPR），Las Vegas，NV，USA，Jun 27-30，2016.Piscataway：IEEE，2016：2536-2544.
[24] IIZUKA S，SIMO-SERRA E，ISHIKAWA H.Globally and locally consistent image completion[J].ACM Transactions on Graphics（ToG），2017，36（4）：1-14.
[25] YU J，LIN Z，YANG J，et al.Generative image inpainting with contextual attention[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition（CVPR），Salt Lake City，UT，USA，June 18-22，2018.Piscataway：IEEE，2018：5505-5514.
[26] ZENG Y，FU J，CHAO H，et al.Learning pyramid-context encoder network for high-quality image inpainting[C]//2019 Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，Long Beach，CA，USA，Jun 15?21，2019.Piscataway：IEEE，2019：1486-1494.
[27] XU S X，LIU D，XIONG Z W.E2I：generative inpainting from edge to image[J].IEEE Transactions on Circuits and Systems for Video Technology，2021，31（4）：1308-1322.
[28] HASSANPOUR A，DARYANI A E，MIRMAHDI M，et al.E2F-GAN：eyes-to-face inpainting via edge-aware coarse-to-fine GANs[J].IEEE Access，2022，10：32406-32417.
[29] 刘颖，张艺轩，佘建初，等.人脸去遮挡新技术研究综述[J].计算机科学与探索，2021，15（10）：1773-1794.
LIU Y，ZHANG Y X，SHE J C，et al.Review of new face occlusion inpainting technology research[J].Journal of Frontiers of Computer Science and Technology，2021，15（10）：1773-1794.
[30] 谌贵辉，陈伍，李忠兵，等.残差卷积注意网络的图像超分辨率重建[J].计算机工程与应用，2021，57（12）：193-200.
SHEN G H，CHEN W，LI Z B，et al.Review of new face occlusion inpainting technology research[J].Computer Engineering and Applications，2021，57（12）：193-200.
[31] ZENG Y H，FU J L，CHAO H Y，et al.Aggregated contextual transformations for high-resolution image inpainting[J].IEEE Transactions on Visualization and Computer Graphics，2022，29：3266-3280.
[32] LIU G，REDA F A，SHIH K J，et al.Image inpainting for irregular holes using partial convolutions[C]//Proceedings of the European Conference on Computer Vision（ECCV），Munich，Germany，2018：85-100.