Research on Image Restoration Method Based on Structure Embedding

doi:10.3778/j.issn.1002-8331.2007-0279

Abstract

Abstract:

In view of the current problem in the field of image repair, there is a problem of lost structure and blurred texture, and it is not possible to make full use of background information to generate a filled area with a consistent content style. Based on the encoder-decoder network, this paper proposes a shared repair model with multi-scale structure information and attention mechanism. In the generation stage, multi-scale structure information is embedded to provide prerequisites for image restoration. At the same time, the multi-scale attention mechanism is used to obtain relevant information on the background information, and refine the content and structure related to the image. This model uses PatchGAN and fixed-weight VGG-16 classifier as the discriminator, and uses style loss and perception loss to the adversarial network in order to achieve the style consistency of the generated images. Compared with the current mainstream image repair algorithms on the Places2 dataset, the results show that proposed the algorithm can restore the detailed information about the image structure better than other algorithms, and generate clearer and more detailed repair results.

Key words: Generative Adversarial Networks（GAN）, attention mechanism, VGG-16, image repair

摘要：

针对当前图像修复领域存在结构丢失、纹理模糊、不能够充分利用背景信息生成内容风格一致的填充区域的问题，在编码解码网络基础上，提出带有多尺度结构信息与注意力机制的共享修复模型。在生成阶段，嵌入多尺度结构信息为图像修复提供前提条件。同时使用多尺度注意力机制，从背景信息中获取相关信息，并经过细化，生成与图像相关的内容和结构；使用PatchGAN和固定权重VGG-16分类器作为鉴别器，并将风格损失和感知损失引入到对抗网络中，以实现所生成图像的风格一致性。在Places2数据集上与当前主流的图像修复算法进行对比，实验结果表明该算法与其他算法相比能较好地恢复图像结构的细节信息，生成更清晰、精细的修复结果。

关键词: 生成对抗网络（GAN）, 注意力机制, VGG-16, 图像修复

WANG Haiyong, LI Haiyang, GAO Xuejiao. Research on Image Restoration Method Based on Structure Embedding[J]. Computer Engineering and Applications, 2021, 57(22): 241-246.

王海涌，李海洋，高雪娇. 基于结构嵌入的图像修复方法研究[J]. 计算机工程与应用, 2021, 57(22): 241-246.

References

[1] ROTH S，BLACK M J.Fields of experts：a framework for learning image priors[C]//2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2005：860-867.
[2] 何雨亭，唐向宏，张越，等.结构张量的改进Criminisi修复[J].中国图象图形学报，2018，23（10）：1492-1507.
HE Y T，TANG X H，ZHANG Y，et al.Improved Criminisi algorithm based on structure tensor[J].Journal of Image and Graphics，2018，23（10）：1492-1507.
[3] PATHAK D，KRAHENBUHL P，DONAHUE J，et al.Context encoders：feature learning by inpainting[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition，2016：2536-2544.
[4] IIZUKA S，SIMO-SERRA E，ISHIKAWA H.Globally and locally consistent image completion[J].ACM Transactions on Graphics，2017，36（4）：107.
[5] YANG C，LU X，LIN Z，et al.High-resolution image inpainting using multi-scale neural patch synthesis[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition，2017：6721-6729.
[6] YU J H，LIN Z，YANG J M，et al.Generative image inpainting with contextual attention[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2018：5505-5514.
[7] NAZERI K，NG E，JOSEPH T，et al.EdgeConnect：generative image inpainting with adversarial edge learning[J].arXiv：1901.00212，2019.
[8] XIONG W，YU J，LIN Z M，et al.Foreground-aware image inpainting[C]//2019 IEEE Conference on Computer Vision and Pattern Recognition，2019：5840-5848.
[9] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial nets[C]//2014 Conference on Computer Vision and Pattern Recognition，2014：2672-2680.
[10] MIRZA M，OSINDERO S.Conditional generative adversarial nets[J].arXiv：1411.1784，2014.
[11] ARJOVSKY M，CHINTALA S，BOTTOU L.Wasserstein GAN[J].arXiv：1701.07875，2017.
[12] GULRAJANI I，AHMED F，ARJOVSKY M，et al.Improved training of Wasserstein GANs[C]//Advances in Neural Information Processing Systems，2017：5767-5777.
[13] ISOLA P，ZHU J，ZHOU T，et al.Image-to-image translation with conditional adversarial networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition，2017：1125-1134.
[14] JADERBERG M，SIMONYAN K，ZISSERMAN A，et al.Spatial transformer networks[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition，2015：2017-2025.
[15] ZHOU T，TULSIANI S，SUN W，et al.View synthesis by appearance flow[C]//14th European Conference on Computer Vision.Cham：Springer，2016：286-301.
[16] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409.
1556，2014.