计算机工程与应用 ›› 2023, Vol. 59 ›› Issue (20): 1-12.DOI: 10.3778/j.issn.1002-8331.2303-0111
吕建峰,邵立珍,雷雪梅
出版日期:
2023-10-15
发布日期:
2023-10-15
LYU Jianfeng, SHAO Lizhen, LEI Xuemei
Online:
2023-10-15
Published:
2023-10-15
摘要: 深度学习的快速发展使计算机视觉技术应用越来越广泛,同时利用深度神经网络根据破损图像的已知信息对图像复原的修复技术成为关注的热点。对近年基于深度神经网络的图像修复方法进行了综述和分析:按照模型优化的方向,对图像修复方法进行分类综述;介绍了图像修复常用的数据集和性能评价指标,并在相关数据集上对各种基于深度神经网络的破损图像修复算法进行性能评价和分析;总结和分析了现有图像修复方法面临的挑战和未来研究方向。
吕建峰, 邵立珍, 雷雪梅. 基于深度神经网络的图像修复算法综述[J]. 计算机工程与应用, 2023, 59(20): 1-12.
LYU Jianfeng, SHAO Lizhen, LEI Xuemei. Image Inpainting Algorithm Based on Deep Neural Networks[J]. Computer Engineering and Applications, 2023, 59(20): 1-12.
[1] JAM J,KENDRICK C,WALKER K,et al.A comprehensive review of past and present image inpainting methods[J].Computer Vision and Image Understanding,2021,203:103-147. [2] KRIZHEVSKY A,SUTSKEVER I,HINTON G E.Imagenet classification with deep convolutional neural networks[J].Communications of the ACM,2017,60(6):84-90. [3] HE K,GKIOXARI G,DOLLáR P,et al.Mask R-CNN[C]//Proceedings of the IEEE International Conference on Computer Vision,2017:2961-2969. [4] ELHARROUSS O,ALMAADEED N,AL-MAADEED S,et al.Image inpainting:a review[J].Neural Processing Letters,2020,51(2):2007-2028. [5] BERTALMIO M,SAPIRO G,CASELLES V,et al.Image inpainting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques,2000:417-424. [6] CHAN T F,SHEN J.Nontexture inpainting by curvature-driven diffusions[J].Journal of Visual Communication and Image Representation,2001,12(4):436-449. [7] CRIMINISI A,PéREZ P,TOYAMA K.Region filling and object removal by exemplar-based image inpainting[J].IEEE Transactions on Image Processing,2004,13(9):1200-1212. [8] BARNES C,SHECHTMAN E,FINKELSTEIN A,et al.PatchMatch:a randomized correspondence algorithm for structural image editing[J].ACM Trans Graph,2009,28(3):24. [9] HAYS J,EFROS A A.Scene completion using millions of photographs[J].Communications of the ACM,2008,51(10):87-94. [10] LECUN Y,BOSER B,DENKER J S,et al.Backpropagation applied to handwritten zip code recognition[J].Neural Computation,1989,1(4):541-551. [11] RUMELHART D E,HINTON G E,WILLIAMS R J.Learning internal representations by error propagation[R].San Diego:California Univ.Inst for Cognitive Science,1985. [12] GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial nets[C]//Advances in Neural Information Processing Systems,2014:2672-2680. [13] 艾亚鹏.基于样本和深度学习的壁画修复研究[D].兰州:兰州交通大学,2021. AI Y P.Research on mural inpainting based on exemplar and deep learning[D].Lanzhou:Lanzhou Jiaotong University,2021. [14] WANG Q,CHEN Y,ZHANG N,et al.Medical image inpainting with edge and structure priors[J].Measurement,2021,185:110027. [15] 刘昱,刘厚泉.基于对抗训练和卷积神经网络的面部图像修复[J].计算机工程与应用,2019,55(2):110-115. LIU Y,LIU H Q.Facial image restoration based on adversarial training and convolutional neural network[J].Computer Engineering and Applications,2019,55(2):110-115. [16] WAN Z,ZHANG B,CHEN D,et al.Bringing old photos back to life[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,2020:2747-2757. [17] HINTON G E,OSINDERO S,TEH Y W.A fast learning algorithm for deep belief nets[J].Neural Computation,2006,18(7):1527-1554. [18] MASCI J,MEIER U,CIRE?AN D,et al.Stacked convolutional auto-encoders for hierarchical feature extraction[C]//International Conference on Artificial Neural Networks.Berlin,Heidelberg:Springer,2011:52-59. [19] PATHAK D,KRAHENBUHL P,DONAHUE J,et al.Context encoders:feature learning by inpainting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,2016:2536-2544. [20] LIU G,REDA F A,SHIH K J,et al.Image inpainting for irregular holes using partial convolutions[C]//Proceedings of the European Conference on Computer Vision(ECCV),2018:85-100. [21] YU J,LIN Z,YANG J,et al.Free-form image inpainting with gated convolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision,2019:4471-4480. [22] WANG Y,TAO X,QI X,et al.Image inpainting via generative multi-column convolutional neural networks[C]//Advances in Neural Information Processing Systems,2018:329-338. [23] ZHANG H,HU Z,LUO C,et al.Semantic image inpainting with progressive generative networks[C]//Proceedings of the 26th ACM International Conference on Multimedia,2018:1939-1947. [24] IOFFE S,SZEGEDY C.Batch normalization:accelerating deep network training by reducing internal covariate shift[C]//International Conference on Machine Learning,2015:448-456. [25] BA J L,KIROS J R,HINTON G E.Layer normalization[J].arXiv:1607.06450,2016. [26] ULYANOV D,VEDALDI A,LEMPITSKY V.Instance normalization:the missing ingredient for fast stylization[J].arXiv:1607.08022,2016. [27] YU T,GUO Z,JIN X,et al.Region normalization for image inpainting[C]//Proceedings of the AAAI Conference on Artificial Intelligence,2020:12733-12740. [28] PENG J,LIU D,XU S,et al.Generating diverse structure for image inpainting with hierarchical VQ-VAE[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,2021:10775-10784. [29] ZENG Y,FU J,CHAO H,et al.Aggregated contextual transformations for high-resolution image inpainting[J].IEEE Transactions on Visualization and Computer Graphics,2023,29(7):3266-3280. [30] QUAN W,ZHANG R,ZHANG Y,et al.Image inpainting with local and global refinement[J].IEEE Transactions on Image Processing,2022,31:2405-2420. [31] ZHENG H,LIN Z,LU J,et al.CM-GAN:image inpainting with cascaded modulation gan and object-aware training[J].arXiv:2203.11947,2022. [32] HE K,CHEN X,XIE S,et al.Masked autoencoders are scalable vision learners[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,2022:16000-16009. [33] ZHANG H,GOODFELLOW I,METAXAS D,et al.Self-attention generative adversarial networks[C]//International Conference on Machine Learning,2019:7354-7363. [34] YAN Z,LI X,LI M,et al.Shift-net:image inpainting via deep feature rearrangement[C]//Proceedings of the European Conference on Computer Vision(ECCV),2018:1-17. [35] RONNEBERGER O,FISCHER P,BROX T.U-NET:convolutional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Computer-assisted Intervention,Munich,Germany,October 5-9,2015.Cham:Springer,2015:234-241. [36] YU J,LIN Z,YANG J,et al.Generative image inpainting with contextual attention[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,2018:5505-5514. [37] LIU H,JIANG B,XIAO Y,et al.Coherent semantic attention for image inpainting[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision,2019:4170-4179. [38] WANG N,LI J,ZHANG L,et al.MUSICAL:multi-scale image contextual attention learning for inpainting[C]//International Joint Conference on Artificial Intelligence,Macao,China,August 10-16,2019:3748-3754. [39] ZENG Y,FU J,CHAO H,et al.Learning pyramid-context encoder network for high-quality image inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,2019:1486-1494. [40] LIN T Y,DOLLáR P,GIRSHICK R,et al.Feature pyramid networks for object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,2017:2117-2125. [41] LI J,WANG N,ZHANG L,et al.Recurrent feature reasoning for image inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,2020:7760-7768. [42] YI Z,TANG Q,AZIZI S,et al.Contextual residual aggregation for ultra high-resolution image inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,2020:7508-7517. [43] VASWANI A,SHAZEER N,PARMAR N,et al.Attention is all you need[C]//Advances in Neural Information Processing Systems,2017. [44] DOSOVITSKIY A,BEYER L,KOLESNIKOV A,et al.An image is worth 16x16 words:transformers for image recognition at scale[J].arXiv:2010.11929,2020. [45] WAN Z,ZHANG J,CHEN D,et al.High-fidelity pluralistic image completion with transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision,Canada,June 19-25,2021:4692-4701. [46] ZHENG C,SONG G,CHAM T J,et al.High-quality pluralistic image completion via code shared VQGAN[J].arXiv:2204.01931,2022. [47] 沈玲.基于语义感知深度模型的图像修复方法研究[D].合肥:合肥工业大学,2020. SHEN L.Research on image inpainting methods based on semantic perception deep model[D].Hefei:Hefei University of Technology,2020. [48] SONG Y,YANG C,SHEN Y,et al.SPG-Net:segmentation prediction and guidance network for image inpainting[J].arXiv:1805.03356,2018. [49] LONG J,SHELHAMER E,DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,2015:3431-3440. [50] NAZERI K,NG E,JOSEPH T,et al.EdgeConnect:structure guided image inpainting using edge prediction[C]//2019 IEEE/CVF International Conference on Computer Vision Workshop(ICCVW),2019:3265-3274. [51] LI J,HE F,ZHANG L,et al.Progressive reconstruction of visual structure for image inpainting[C]//2019 IEEE/CVF International Conference on Computer Vision(ICCV),2019:5961-5970. [52] LIAO L,XIAO J,WANG Z,et al.Guidance and evaluation:semantic-aware image inpainting for mixed scenes[C]//Computer Vision-ECCV 2020,2020:683-700. [53] LIU Y,PAN J,SU Z.Deep blind image inpainting[C]//International Conference on Intelligent Science and Big Data Engineering.Cham:Springer,2019:128-141. [54] WANG Y,CHEN Y C,TAO X,et al.VCNet:a robust approach to blind image inpainting[C]//Computer Vision-ECCV 2020,2020:752-768. [55] WANG T,OUYANG H,CHEN Q.Image inpainting with external-internal learning and monochromic bottleneck[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,Kuala Lumpur,Malaysia,December 18-20,2021:5120-5129. [56] ZENG Y,LIN Z,YANG J,et al.High-resolution image inpainting with iterative confidence feedback and guided upsampling[C]//European Conference on Computer Vision,Glasgow,UK,August 23-28,2020.Cham:Springer,2020:1-17. [57] GUO X,YANG H,HUANG D.Image inpainting via conditional texture and structure dual generation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision,Canada,June 19-25,2021:14134-14143. [58] IIZUKA S,SIMO-SERRA E,ISHIKAWA H.Globally and locally consistent image completion[J].ACM Transactions on Graphics(ToG),2017,36(4):1-14. [59] JOHNSON J,ALAHI A,FEI-FEI L.Perceptual losses for real-time style transfer and super-resolution[C]//Computer Vision-ECCV 2016:14th European Conference,Amsterdam,The Netherlands,October 11-14,2016:694-711. [60] 李月龙,高云,闫家良,等.基于深度神经网络的图像缺损修复方法综述[J].计算机学报,2021,44(11):2295-2316. LI Y L,GAO Y,YAN J L,et al.Image inpainting methods based on deep neural networks:a review[J].Chinese Journal of Computers,2021,44(11):2295-2316. [61] YANG C,LU X,LIN Z,et al.High-resolution image inpainting using multi-scale neural patch synthesis[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,2017:6721-6729. [62] SIMONYAN K,ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv:1409. 1556,2014. [63] ISOLA P,ZHU J Y,ZHOU T,et al.Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,2017:1125-1134. [64] MIYATO T,KATAOKA T,KOYAMA M,et al.Spectral normalization for generative adversarial networks[J].arXiv:1802.05957,2018. [65] ZENG Y,LIN Z,LU H,et al.CR-Fill:generative image inpainting with auxiliary contextual reconstruction[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision,Canada,June 19-25,2021:14164-14173. [66] DOERSCH C,SINGH S,GUPTA A,et al.What makes paris look like paris?[J].ACM Transactions on Graphics,2012,31(4):1-9. [67] CORDTS M,OMRAN M,RAMOS S,et al.The cityscapes dataset for semantic urban scene understanding[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,2016:3213-3223. [68] ZHOU B,LAPEDRIZA A,KHOSLA A,et al.Places:a 10 million image database for scene recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence,2017,40(6):1452-1464. [69] RUSSAKOVSKY O,DENG J,SU H,et al.Imagenet large scale visual recognition challenge[J].International Journal of Computer Vision,2015,115:211-252. [70] LIU Z,LUO P,WANG X,et al.Deep learning face attributes in the wild[C]//Proceedings of the IEEE International Conference on Computer Vision,Santiago,Chile,December 13-16,2015.New York:IEEE,2015:3730-3738. [71] KARRAS T,AILA T,LAINE S,et al.Progressive growing of GANs for improved quality,stability,and variation[J].arXiv:1710.10196,2017. [72] KARRAS T,LAINE S,AILA T.A style-based generator architecture for generative adversarial networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,2019:4401-4410. [73] ISKAKOV K.Semi-parametric image inpainting[J].arXiv:1807.02855,2018. [74] WANG Z,BOVIK A C,SHEIKH H R,et al.Image quality assessment:from error visibility to structural similarity[J].IEEE Transactions on Image Processing,2004,13(4):600-612. [75] HEUSEL M,RAMSAUER H,UNTERTHINER T,et al.Gans trained by a two time-scale update rule converge to a local nash equilibrium[C]//Advances in Neural Information Processing Systems,2017. [76] SZEGEDY C,VANHOUCKE V,IOFFE S,et al.Rethinking the inception architecture for computer vision[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition,2016:2818-2826. |
[1] | 崔少国, 独潇, 杨泽田. 多注意力机制融合低高阶特征的神经推荐算法[J]. 计算机工程与应用, 2023, 59(8): 192-199. |
[2] | 梁义涛, 韩永波, 李磊. 深度长时目标跟踪算法综述[J]. 计算机工程与应用, 2023, 59(4): 1-17. |
[3] | 方祯煜, 尹海涛. 面向图像去噪的深度双层群稀疏编码网络[J]. 计算机工程与应用, 2023, 59(21): 195-203. |
[4] | 彭豪, 李晓明. 结合特征调整与联合自注意力的图像修复[J]. 计算机工程与应用, 2023, 59(19): 184-191. |
[5] | 刘梦庭, 凌捷. 优化梯度增强黑盒对抗攻击算法[J]. 计算机工程与应用, 2023, 59(18): 260-267. |
[6] | 刘绪颖, 卢文达, 王剑, 王雪, 王庆. 融合多变量序列时空信息的事件早期识别方法[J]. 计算机工程与应用, 2023, 59(17): 116-122. |
[7] | 姚光乐, 祝钧桃, 周文龙, 张贵宇, 张伟, 张谦. 基于特征分布学习的小样本类增量学习[J]. 计算机工程与应用, 2023, 59(14): 151-157. |
[8] | 王少华, 刘法胜, 时柏营, 刘兴波, 聂秀山. 面向交通标识的二值语义嵌入学习方法[J]. 计算机工程与应用, 2023, 59(13): 205-210. |
[9] | 刘金琳, 李冬冬, 王喆, 蔡立志. 两级特征联合学习的情感说话人识别[J]. 计算机工程与应用, 2023, 59(1): 149-155. |
[10] | 杨曦, 闫杰, 王文, 李少毅, 林健. 脑启发的视觉目标识别模型研究与展望[J]. 计算机工程与应用, 2022, 58(7): 1-20. |
[11] | 马幪浩, 王喆. 小样本下基于Wasserstein距离的半监督学习算法[J]. 计算机工程与应用, 2022, 58(5): 193-199. |
[12] | 贠璟扬, 李学华, 向维. 语义导向多尺度多视图深度估计算法[J]. 计算机工程与应用, 2022, 58(2): 215-224. |
[13] | 王瑞平, 吴士泓, 张美航, 王小平. 视觉问答语言处理方法综述[J]. 计算机工程与应用, 2022, 58(17): 50-60. |
[14] | 霍相佐, 张文东, 田生伟, 侯树祥. 面向边-端协同的并行解码器图像修复方法[J]. 计算机工程与应用, 2022, 58(16): 257-264. |
[15] | 于强, 林民, 李艳玲. 基于深度学习的关键词生成研究综述[J]. 计算机工程与应用, 2022, 58(14): 27-39. |
阅读次数 | ||||||
全文 |
|
|||||
摘要 |
|
|||||