Image Inpainting Algorithm Based on Deep Neural Networks

doi:10.3778/j.issn.1002-8331.2303-0111

Abstract

Abstract: With the rapid development of deep learning, computer vision technology is applied more and more widely. At the same time, the image inpainting technology based on the known information of the damaged image using deep neural network has also become a hot topic. The image inpainting methods based on depth neural network in recent years are reviewed and analyzed. Firstly, the image inpainting methods are classified and summarized according to the view of model optimization. Then the common datasets and performance evaluation indicators are introduced, and the performance evaluation and analysis of various deep neural network-based image inpainting algorithms are carried out on the relevant data sets. Finally, the challenges faced by the existing image inpainting methods are analyzed, and the future research works are prospected.

Key words: deep neural networks, image inpainting, algorithm analysis

摘要： 深度学习的快速发展使计算机视觉技术应用越来越广泛，同时利用深度神经网络根据破损图像的已知信息对图像复原的修复技术成为关注的热点。对近年基于深度神经网络的图像修复方法进行了综述和分析：按照模型优化的方向，对图像修复方法进行分类综述；介绍了图像修复常用的数据集和性能评价指标，并在相关数据集上对各种基于深度神经网络的破损图像修复算法进行性能评价和分析；总结和分析了现有图像修复方法面临的挑战和未来研究方向。

关键词: 深度神经网络, 图像修复, 算法分析

LYU Jianfeng, SHAO Lizhen, LEI Xuemei. Image Inpainting Algorithm Based on Deep Neural Networks[J]. Computer Engineering and Applications, 2023, 59(20): 1-12.

吕建峰, 邵立珍, 雷雪梅. 基于深度神经网络的图像修复算法综述[J]. 计算机工程与应用, 2023, 59(20): 1-12.

References

[1] JAM J，KENDRICK C，WALKER K，et al.A comprehensive review of past and present image inpainting methods[J].Computer Vision and Image Understanding，2021，203：103-147.
[2] KRIZHEVSKY A，SUTSKEVER I，HINTON G E.Imagenet classification with deep convolutional neural networks[J].Communications of the ACM，2017，60（6）：84-90.
[3] HE K，GKIOXARI G，DOLLáR P，et al.Mask R-CNN[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2961-2969.
[4] ELHARROUSS O，ALMAADEED N，AL-MAADEED S，et al.Image inpainting：a review[J].Neural Processing Letters，2020，51（2）：2007-2028.
[5] BERTALMIO M，SAPIRO G，CASELLES V，et al.Image inpainting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques，2000：417-424.
[6] CHAN T F，SHEN J.Nontexture inpainting by curvature-driven diffusions[J].Journal of Visual Communication and Image Representation，2001，12（4）：436-449.
[7] CRIMINISI A，PéREZ P，TOYAMA K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing，2004，13（9）：1200-1212.
[8] BARNES C，SHECHTMAN E，FINKELSTEIN A，et al.PatchMatch：a randomized correspondence algorithm for structural image editing[J].ACM Trans Graph，2009，28（3）：24.
[9] HAYS J，EFROS A A.Scene completion using millions of photographs[J].Communications of the ACM，2008，51（10）：87-94.
[10] LECUN Y，BOSER B，DENKER J S，et al.Backpropagation applied to handwritten zip code recognition[J].Neural Computation，1989，1（4）：541-551.
[11] RUMELHART D E，HINTON G E，WILLIAMS R J.Learning internal representations by error propagation[R].San Diego：California Univ.Inst for Cognitive Science，1985.
[12] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial nets[C]//Advances in Neural Information Processing Systems，2014：2672-2680.
[13] 艾亚鹏.基于样本和深度学习的壁画修复研究[D].兰州：兰州交通大学，2021.
AI Y P.Research on mural inpainting based on exemplar and deep learning[D].Lanzhou：Lanzhou Jiaotong University，2021.
[14] WANG Q，CHEN Y，ZHANG N，et al.Medical image inpainting with edge and structure priors[J].Measurement，2021，185：110027.
[15] 刘昱，刘厚泉.基于对抗训练和卷积神经网络的面部图像修复[J].计算机工程与应用，2019，55（2）：110-115.
LIU Y，LIU H Q.Facial image restoration based on adversarial training and convolutional neural network[J].Computer Engineering and Applications，2019，55（2）：110-115.
[16] WAN Z，ZHANG B，CHEN D，et al.Bringing old photos back to life[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：2747-2757.
[17] HINTON G E，OSINDERO S，TEH Y W.A fast learning algorithm for deep belief nets[J].Neural Computation，2006，18（7）：1527-1554.
[18] MASCI J，MEIER U，CIRE?AN D，et al.Stacked convolutional auto-encoders for hierarchical feature extraction[C]//International Conference on Artificial Neural Networks.Berlin，Heidelberg：Springer，2011：52-59.
[19] PATHAK D，KRAHENBUHL P，DONAHUE J，et al.Context encoders：feature learning by inpainting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：2536-2544.
[20] LIU G，REDA F A，SHIH K J，et al.Image inpainting for irregular holes using partial convolutions[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：85-100.
[21] YU J，LIN Z，YANG J，et al.Free-form image inpainting with gated convolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：4471-4480.
[22] WANG Y，TAO X，QI X，et al.Image inpainting via generative multi-column convolutional neural networks[C]//Advances in Neural Information Processing Systems，2018：329-338.
[23] ZHANG H，HU Z，LUO C，et al.Semantic image inpainting with progressive generative networks[C]//Proceedings of the 26th ACM International Conference on Multimedia，2018：1939-1947.
[24] IOFFE S，SZEGEDY C.Batch normalization：accelerating deep network training by reducing internal covariate shift[C]//International Conference on Machine Learning，2015：448-456.
[25] BA J L，KIROS J R，HINTON G E.Layer normalization[J].arXiv：1607.06450，2016.
[26] ULYANOV D，VEDALDI A，LEMPITSKY V.Instance normalization：the missing ingredient for fast stylization[J].arXiv：1607.08022，2016.
[27] YU T，GUO Z，JIN X，et al.Region normalization for image inpainting[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2020：12733-12740.
[28] PENG J，LIU D，XU S，et al.Generating diverse structure for image inpainting with hierarchical VQ-VAE[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：10775-10784.
[29] ZENG Y，FU J，CHAO H，et al.Aggregated contextual transformations for high-resolution image inpainting[J].IEEE Transactions on Visualization and Computer Graphics，2023，29（7）：3266-3280.
[30] QUAN W，ZHANG R，ZHANG Y，et al.Image inpainting with local and global refinement[J].IEEE Transactions on Image Processing，2022，31：2405-2420.
[31] ZHENG H，LIN Z，LU J，et al.CM-GAN：image inpainting with cascaded modulation gan and object-aware training[J].arXiv：2203.11947，2022.
[32] HE K，CHEN X，XIE S，et al.Masked autoencoders are scalable vision learners[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：16000-16009.
[33] ZHANG H，GOODFELLOW I，METAXAS D，et al.Self-attention generative adversarial networks[C]//International Conference on Machine Learning，2019：7354-7363.
[34] YAN Z，LI X，LI M，et al.Shift-net：image inpainting via deep feature rearrangement[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：1-17.
[35] RONNEBERGER O，FISCHER P，BROX T.U-NET：convolutional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Computer-assisted Intervention，Munich，Germany，October 5-9，2015.Cham：Springer，2015：234-241.
[36] YU J，LIN Z，YANG J，et al.Generative image inpainting with contextual attention[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：5505-5514.
[37] LIU H，JIANG B，XIAO Y，et al.Coherent semantic attention for image inpainting[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：4170-4179.
[38] WANG N，LI J，ZHANG L，et al.MUSICAL：multi-scale image contextual attention learning for inpainting[C]//International Joint Conference on Artificial Intelligence，Macao，China，August 10-16，2019：3748-3754.
[39] ZENG Y，FU J，CHAO H，et al.Learning pyramid-context encoder network for high-quality image inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：1486-1494.
[40] LIN T Y，DOLLáR P，GIRSHICK R，et al.Feature pyramid networks for object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：2117-2125.
[41] LI J，WANG N，ZHANG L，et al.Recurrent feature reasoning for image inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：7760-7768.
[42] YI Z，TANG Q，AZIZI S，et al.Contextual residual aggregation for ultra high-resolution image inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：7508-7517.
[43] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems，2017.
[44] DOSOVITSKIY A，BEYER L，KOLESNIKOV A，et al.An image is worth 16x16 words：transformers for image recognition at scale[J].arXiv：2010.11929，2020.
[45] WAN Z，ZHANG J，CHEN D，et al.High-fidelity pluralistic image completion with transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，Canada，June 19-25，2021：4692-4701.
[46] ZHENG C，SONG G，CHAM T J，et al.High-quality pluralistic image completion via code shared VQGAN[J].arXiv：2204.01931，2022.
[47] 沈玲.基于语义感知深度模型的图像修复方法研究[D].合肥：合肥工业大学，2020.
SHEN L.Research on image inpainting methods based on semantic perception deep model[D].Hefei：Hefei University of Technology，2020.
[48] SONG Y，YANG C，SHEN Y，et al.SPG-Net：segmentation prediction and guidance network for image inpainting[J].arXiv：1805.03356，2018.
[49] LONG J，SHELHAMER E，DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：3431-3440.
[50] NAZERI K，NG E，JOSEPH T，et al.EdgeConnect：structure guided image inpainting using edge prediction[C]//2019 IEEE/CVF International Conference on Computer Vision Workshop（ICCVW），2019：3265-3274.
[51] LI J，HE F，ZHANG L，et al.Progressive reconstruction of visual structure for image inpainting[C]//2019 IEEE/CVF International Conference on Computer Vision（ICCV），2019：5961-5970.
[52] LIAO L，XIAO J，WANG Z，et al.Guidance and evaluation：semantic-aware image inpainting for mixed scenes[C]//Computer Vision-ECCV 2020，2020：683-700.
[53] LIU Y，PAN J，SU Z.Deep blind image inpainting[C]//International Conference on Intelligent Science and Big Data Engineering.Cham：Springer，2019：128-141.
[54] WANG Y，CHEN Y C，TAO X，et al.VCNet：a robust approach to blind image inpainting[C]//Computer Vision-ECCV 2020，2020：752-768.
[55] WANG T，OUYANG H，CHEN Q.Image inpainting with external-internal learning and monochromic bottleneck[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，Kuala Lumpur，Malaysia，December 18-20，2021：5120-5129.
[56] ZENG Y，LIN Z，YANG J，et al.High-resolution image inpainting with iterative confidence feedback and guided upsampling[C]//European Conference on Computer Vision，Glasgow，UK，August 23-28，2020.Cham：Springer，2020：1-17.
[57] GUO X，YANG H，HUANG D.Image inpainting via conditional texture and structure dual generation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，Canada，June 19-25，2021：14134-14143.
[58] IIZUKA S，SIMO-SERRA E，ISHIKAWA H.Globally and locally consistent image completion[J].ACM Transactions on Graphics（ToG），2017，36（4）：1-14.
[59] JOHNSON J，ALAHI A，FEI-FEI L.Perceptual losses for real-time style transfer and super-resolution[C]//Computer Vision-ECCV 2016：14th European Conference，Amsterdam，The Netherlands，October 11-14，2016：694-711.
[60] 李月龙，高云，闫家良，等.基于深度神经网络的图像缺损修复方法综述[J].计算机学报，2021，44（11）：2295-2316.
LI Y L，GAO Y，YAN J L，et al.Image inpainting methods based on deep neural networks：a review[J].Chinese Journal of Computers，2021，44（11）：2295-2316.
[61] YANG C，LU X，LIN Z，et al.High-resolution image inpainting using multi-scale neural patch synthesis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：6721-6729.
[62] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409.
1556，2014.
[63] ISOLA P，ZHU J Y，ZHOU T，et al.Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：1125-1134.
[64] MIYATO T，KATAOKA T，KOYAMA M，et al.Spectral normalization for generative adversarial networks[J].arXiv：1802.05957，2018.
[65] ZENG Y，LIN Z，LU H，et al.CR-Fill：generative image inpainting with auxiliary contextual reconstruction[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，Canada，June 19-25，2021：14164-14173.
[66] DOERSCH C，SINGH S，GUPTA A，et al.What makes paris look like paris?[J].ACM Transactions on Graphics，2012，31（4）：1-9.
[67] CORDTS M，OMRAN M，RAMOS S，et al.The cityscapes dataset for semantic urban scene understanding[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：3213-3223.
[68] ZHOU B，LAPEDRIZA A，KHOSLA A，et al.Places：a 10 million image database for scene recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，40（6）：1452-1464.
[69] RUSSAKOVSKY O，DENG J，SU H，et al.Imagenet large scale visual recognition challenge[J].International Journal of Computer Vision，2015，115：211-252.
[70] LIU Z，LUO P，WANG X，et al.Deep learning face attributes in the wild[C]//Proceedings of the IEEE International Conference on Computer Vision，Santiago，Chile，December 13-16，2015.New York：IEEE，2015：3730-3738.
[71] KARRAS T，AILA T，LAINE S，et al.Progressive growing of GANs for improved quality，stability，and variation[J].arXiv：1710.10196，2017.
[72] KARRAS T，LAINE S，AILA T.A style-based generator architecture for generative adversarial networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：4401-4410.
[73] ISKAKOV K.Semi-parametric image inpainting[J].arXiv：1807.02855，2018.
[74] WANG Z，BOVIK A C，SHEIKH H R，et al.Image quality assessment：from error visibility to structural similarity[J].IEEE Transactions on Image Processing，2004，13（4）：600-612.
[75] HEUSEL M，RAMSAUER H，UNTERTHINER T，et al.Gans trained by a two time-scale update rule converge to a local nash equilibrium[C]//Advances in Neural Information Processing Systems，2017.
[76] SZEGEDY C，VANHOUCKE V，IOFFE S，et al.Rethinking the inception architecture for computer vision[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition，2016：2818-2826.