Computer Engineering and Applications ›› 2024, Vol. 60 ›› Issue (22): 58-73.DOI: 10.3778/j.issn.1002-8331.2406-0100
• Research Hotspots and Reviews • Previous Articles Next Articles
CHEN Wenxiang, TIAN Qichuan, LIAN Lu, ZHANG Xiaohang, WANG Haoji
Online:
2024-11-15
Published:
2024-11-14
陈文祥,田启川,廉露,张晓行,王浩吉
CHEN Wenxiang, TIAN Qichuan, LIAN Lu, ZHANG Xiaohang, WANG Haoji. Research Progress of Image Inpainting Methods Based on Deep Learning[J]. Computer Engineering and Applications, 2024, 60(22): 58-73.
陈文祥, 田启川, 廉露, 张晓行, 王浩吉. 基于深度学习的图像修复方法研究进展[J]. 计算机工程与应用, 2024, 60(22): 58-73.
Add to citation manager EndNote|Ris|BibTeX
URL: http://cea.ceaj.org/EN/10.3778/j.issn.1002-8331.2406-0100
[1] 罗海银, 郑钰辉. 图像修复方法研究综述[J]. 计算机科学与探索, 2022, 16(10): 2193-2218. LUO H Y, ZHENG Y H. Survey of research on image inpainting methods[J]. Journal of Frontiers of Computer Science and Technology, 2022, 16(10): 2193-2218. [2] WAN Z, ZHANG B, CHEN D, et al. Old photo restoration via deep latent space translation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(2): 2071-2087. [3] 刘继鑫, 陈瑞, 安仕鹏. 融合参考先验与生成先验的老照片修复[J]. 中国图象图形学报, 2022, 27(5): 1657-1668. LIU J X, CHEN R, AN S P. Reference prior and generative prior linked distorted old photos restoration[J]. Journal of Image and Graphics, 2022, 27(5): 1657-1668. [4] WU Z, XUAN H, SUN C, et al. Semi-supervised video inpainting with cycle consistency constraints[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 22586-22595. [5] WU J, LI X, SI C, et al. Towards language-driven video inpainting via multimodal large language models[C]//Pro-ceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024: 12501-12511. [6] WEI T T, KUO C, TSENG Y C, et al. MPVF: 4D medical image inpainting by multi-pyramid voxel flows[J]. IEEE Journal of Biomedical and Health Informatics, 2023, 27(12): 5872-5882. [7] SHOBI V M, DHANASEELAN F R. Voxel representation of brain images inpainting via regional pixel semantic network and pyramidal attention ae-quantile differential mechanism model[J]. Computers in Biology and Medicine, 2024, 170: 107767. [8] 吕建峰, 邵立珍, 雷雪梅. 基于深度神经网络的图像修复算法综述[J]. 计算机工程与应用, 2023, 59(20): 1-12. LYU J F, SHAO L Z, LEI X M. Image inpainting algorithm based on deep neural networks[J]. Computer Engineering and Applications, 2023, 59(20): 1-12. [9] LECUN Y, BOSER B, DENKER J S, et al. Back-propagation applied to handwritten zip code recognition[J]. Neural Computation, 1989, 1(4): 541-551. [10] RUMELHART D E, HINTON G E, WILLIAMS R J. Learning internal representations by error propagation[R]. San Diego La Jolla: California University. Institute for Cognitive Science, 1985. [11] PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, 2016: 2536-2544. [12] IIZUKA S, SIMO-SERRA E, ISHIKAWA H. Globally and locally consistent image completion[J]. ACM Transactions on Graphics, 2017, 36(4): 1-14. [13] YANG C, LU X, LIN Z, et al. High-resolution image inpainting using multi-scale neural patch synthesis[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, 2017: 6721-6729. [14] GATYS L A, ECKER A S, BETHGE M. A neural algorithm of artistic style[J]. arXiv:1508.06576, 2015. [15] WANG Y, TAO X, QI X, et al. Image inpainting via generative multi-column convolutional neural networks[C]//Advances in Neural Information Processing Systems 31, 2018. [16] YAN Z, LI X, LI M, et al. Shift-net: image inpainting via deep feature rearrangement[C]//Proceedings of the 15th European Conference on Computer Vision, 2018: 1-17. [17] YANG J, QI Z, SHI Y. Learning to incorporate structure knowledge for image inpainting[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence, 2020: 12605-12612. [18] SHAO X, YE H, YANG B, et al. Two-stream coupling network with bidirectional interaction between structure and texture for image inpainting[J]. Expert Systems with Applications, 2023, 231: 120700. [19] ZHANG Y, LIU Y, HU R, et al. Mutual dual-task generator with adaptive attention fusion for image inpainting[J]. IEEE Transactions on Multimedia, 2024, 26: 1539-1550. [20] LIU G, REDA F A, SHIH K J, et al. Image inpainting for irregular holes using partial convolutions[C]//Proceedings of the 15th European Conference on Computer Vision, 2018: 85-100. [21] YU J, LIN Z, YANG J, et al. Free-form image inpainting with gated convolution[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 3, 2019. Piscataway: IEEE, 2019: 4470-4479. [22] YU X, XU L, LI J, et al. MAGConv: mask-guided convolution for image inpainting[J]. IEEE Transactions on Image Processing, 2023, 32: 4716-4727. [23] DENG Y, HUI S, ZHOU S, et al. Context adaptive network for image inpainting[J]. IEEE Transactions on Image Processing, 2023, 32: 6332-6345. [24] LIU C, XU S, PENG J, et al. Towards interactive image inpainting via robust sketch refinement[J]. IEEE Transactions on Multimedia, 2024. DOI:10.1109/TMM.2024.3402620. [25] XIANG H, MIN W, HAN Q, et al. Structure-aware multi-view image inpainting using dual consistency attention[J]. Information Fusion, 2024, 104: 102174. [26] 樊瑶, 石英男, 柏劲咸. 基于边缘与注意力跨层转移的图像修复模型[J]. 计算机工程, 2023, 49(6): 180-192. FAN Y, SHI Y N, BAI J X. Image inpainting model based on edge and attention transfer across layers[J]. Computer Engineering, 2023, 49(6): 180-192. [27] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Advances in Neural Information Processing Systems 27, 2014. [28] 龚颖, 许文韬, 赵策, 等. 生成对抗网络在图像修复中的应用综述[J]. 计算机科学与探索, 2024, 18(3): 553-573. GONG Y, XU W T, ZHAO C, et al. Review of application of generative adversarial networks in image restoration[J]. Journal of Frontiers of Computer Science and Technology, 2024, 18(3): 553-573. [29] ZHANG X, WANG X, SHI C, et al. DE-GAN: domain embedded GAN for high quality face image inpainting[J]. Pattern Recognition, 2022, 124: 108415. [30] KINGMA D P, WELLING M. Auto-encoding variational bayes[J]. arXiv:1312.6114, 2013. [31] NAZERI K, NG E, JOSEPH T, et al. EdgeConnect: generative image inpainting with adversarial edge learning[J]. arXiv:1901.00212, 2019. [32] TIAN H, ZHANG L, LI S, et al. Pyramid-VAE-GAN: transferring hierarchical latent variables for image inpainting[J]. Computational Visual Media, 2023, 9(4): 827-841. [33] LI H, LI G, LIN L, et al. Context-aware semantic inpainting[J]. IEEE Transactions on Cybernetics, 2018, 49(12): 4398-4411. [34] YU J, LIN Z, YANG J, et al. Generative image inpainting with contextual attention[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, 2018: 5505-5514. [35] 陈晓雷, 杨佳, 梁其铎. 结合语义先验和深度注意力残差的图像修复[J]. 计算机科学与探索, 2023, 17(10): 2450-2461. CHEN X L, YANG J, LIANG Q D. Image inpainting combining semantic priors and deep attention residuals[J]. Journal of Frontiers of Computer Science and Technology, 2023, 17(10): 2450-2461. [36] ZENG Y, FU J, CHAO H, et al. Aggregated contextual transformations for high-resolution image inpainting[J]. IEEE Transactions on Visualization and Computer Graphics, 2023, 29(7): 3266-3280. [37] CAO L, YANG T, WANG Y, et al. Generator pyramid for high-resolution image inpainting[J]. Complex & Intelligent Systems, 2023, 9(6): 6297-6306. [38] CHEN M, ZANG S, AI Z, et al. RFA-Net: residual feature attention network for fine-grained image inpainting[J]. Engineering Applications of Artificial Intelligence, 2023, 119: 105814. [39] LI X, WANG Z, CHEN C, et al. SemID: blind image inpainting with semantic inconsistency detection[J]. Tsinghua Science and Technology, 2024, 29(4): 1053-1068. [40] 周遵富, 张乾, 李伟, 等. 基于高效注意力模块的三阶段网络图像修复[J]. 软件导刊, 2023, 22(8): 196-202. ZHOU Z F, ZHANG Q, LI W, et al. Three-stage network image inpainting based on efficient attention module[J]. Software Guide, 2023, 22(8): 196-202. [41] CHEN Y, XIA R, YANG K, et al. DNNAM: image inpainting algorithm via deep neural networks and attention mechanism[J]. Applied Soft Computing, 2024, 154: 111392. [42] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems 30, 2017. [43] DONG Q, CAO C, FU Y. Incremental transformer structure enhanced image inpainting with masking positional enco-ding[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 11358-11368. [44] CAO C, DONG Q, FU Y. ZITS++: image inpainting by improving the incremental transformer on structural priors[J]. IEEE Transactions on Pattern Analysis and Machine Inte-lligence, 2023, 45(10): 12667-12684. [45] MIAO W, WANG L, LU H, et al. ITrans: generative image inpainting with transformers[J]. Multimedia Systems, 2024, 30(1): 21. [46] LIU J, GONG M, GAO Y, et al. Bidirectional interaction of CNN and transformer for image inpainting[J]. Knowledge-Based Systems, 2024, 299: 112046. [47] DENG Y, HUI S, ZHOU S, et al. T-former: an efficient trans-former for image inpainting[C]//Proceedings of the 30th ACM International Conference on Multimedia, 2022: 6559-6568. [48] CAMPANA J L F, DECKER L G L, E SOUZA M R, et al. Variable-hyperparameter visual transformer for efficient image inpainting[J]. Computers & Graphics, 2023, 113: 57-68. [49] HUANG W, DENG Y, HUI S, et al. Sparse self-attention transformer for image inpainting[J]. Pattern Recognition, 2024, 145: 109897. [50] LI W, LIN Z, ZHOU K, et al. MAT: mask-aware transformer for large hole image inpainting[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 10758-10768. [51] SHAMSOLMOALI P, ZAREAPOOR M, GRANGER E. TransInpaint: transformer-based image inpainting with context adaptation[C]//Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision, 2023: 849-858. [52] WAN Z, ZHANG J, CHEN D, et al. High-fidelity pluralistic image completion with transformers[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision, 2021: 4692-4701. [53] LIU Q, TAN Z, CHEN D, et al. Reduce information loss in transformers for pluralistic image inpainting[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 11347-11357. [54] HO J, JAIN A, ABBEEL P. Denoising diffusion probabilistic models[C]//Advances in Neural Information Processing Systems 33, 2020: 6840-6851. [55] LUGMAYR A, DANELLJAN M, ROMERO A, et al. RePaint: inpainting using denoising diffusion probabilistic models [C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 11461-11471. [56] LIU H, WANG Y, QIAN B, et al. Structure matters: tackling the semantic discrepancy in diffusion models for image inpainting[J]. arXiv:2403.19898, 2024. [57] GRECHKA A, COUAIRON G, CORD M. GradPaint: gradient-guided inpainting with diffusion models[J]. Computer Vision and Image Understanding, 2024, 240: 103928. [58] ZHANG G, JI J, ZHANG Y, et al. Towards coherent image inpainting using denoising diffusion implicit models[J]. arXiv:2304.03322, 2023. [59] ROMBACH R, BLATTMANN A, LORENZ D, et al. High-resolution image synthesis with latent diffusion models[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 10684-10695. [60] XIA B, ZHANG Y, WANG S, et al. DiffIR: efficient diffusion model for image restoration[C]//Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision, 2023: 13095-13105. [61] HUANG Y, HUANG J, LIU J, et al. WaveDM: wavelet-based diffusion models for image restoration[J]. IEEE Transactions on Multimedia, 2024, 26: 7058-7073. [62] CORNEANU C, GADDE R, MARTINEZ A M. LatentPaint: image inpainting in latent space with diffusion models[C]//Proceedings of the 2024 IEEE/CVF Winter Conference on Applications of Computer Vision, 2024: 4334-4343. [63] ZHANG L, CHEN Q, HU B, et al. Text-guided neural image inpainting[C]//Proceedings of the 28th ACM International Conference on Multimedia, 2020: 1302-1310. [64] LIN Q, YAN B, LI J, et al. MMFL: multimodal fusion learning for text-guided image inpainting[C]//Proceedings of the 28th ACM International Conference on Multimedia, 2020: 1094-1102. [65] XIE Y, LIN Z, YANG Z, et al. Learning semantic alignment from image for text-guided image inpainting[J]. The Visual Computer, 2022, 38(9): 3149-3161. [66] LI A, ZHAO L, ZUO Z, et al. MIGT: multi-modal image inpainting guided with text[J]. Neurocomputing, 2023, 520: 376-385. [67] WU X, ZHAO K, HUANG Q, et al. MISL: multi-grained image-text semantic learning for text-guided image inpainting[J]. Pattern Recognition, 2024, 145: 109961. [68] ZHAN D, WU J, LUO X, et al. Learning from text: a multi-modal face inpainting network for irregular holes[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34(8): 7484-7497. [69] MEISHVILI G, JENNI S, FAVARO P. Learning to have an ear for face super-resolution[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 1361-1371. [70] KOUMPAROULIS A, POTAMIANOS G, THOMAS S, et al. Audio-assisted image inpainting for talking faces[C]//Proceedings of the 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, 2020: 7664-7668. [71] KIM K, JUNG J, KIM W J, et al. Deep video inpainting guided by audio-visual self-supervision[C]//Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing, 2022: 1970-1974. [72] SANGUINETI V, THAKUR S, MORERIO P, et al. Audio-visual inpainting: reconstructing missing visual information with sound[C]//Proceedings of the 2023 IEEE International Conference on Acoustics, Speech and Signal Processing, 2023: 1-5. [73] ZHANG Z, WU B, WANG X, et al. AVID: any-length video inpainting with diffusion model[C]//Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024: 7162-7172. [74] GU B, YU Y, FAN H, et al. Flow-guided diffusion for video inpainting[J]. arXiv:2311.15368, 2023. [75] ZHOU S, LI C, CHAN K C K, et al. ProPainter: improving propagation and transformer for video inpainting[C]//Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision, 2023: 10477-10486. [76] LIU R, ZHU Y. FSTT: flow-guided spatial temporal transformer for deep video inpainting[J]. Electronics, 2023, 12(21): 4452. [77] LEE E, YOO J, YANG Y, et al. Semantic-aware dynamic parameter for video inpainting transformer[C]//Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision, 2023: 12949-12958. [78] ZHANG C, YANG W, LI X, et al. MMGInpainting: multi-modality guided image inpainting based on diffusion models[J]. IEEE Transactions on Multimedia, 2024, 26: 8811-8823. [79] CHEN C, NISHIO T, BENNIS M, et al. Rf-Inpainter: multimodal image inpainting based on vision and radio signals [J]. IEEE Access, 2022, 10: 110689-110700. [80] XU J, GANDELSMAN Y, BAR A, et al. IMProv: inpainting-based multimodal prompting for computer vision tasks[J]. arXiv:2312.01771, 2023. [81] YANG S, CHEN X, LIAO J. Uni-paint: a unified framework for multimodal image inpainting with pretrained diffusion model[C]//Proceedings of the 31st ACM International Conference on Multimedia, 2023: 3190-3199. [82] YU Y, DU D, ZHANG L, et al. Unbiased multi-modality guidance for image inpainting[C]//Proceedings of the 17th European Conference on Computer Vision. Cham: Springer, 2022: 668-684. [83] WANG H, YU Y, LUO T, et al. MaGIC: multi-modality guided image completion[C]//Proceedings of the 12th International Conference on Learning Representations, 2024. [84] ISKAKOV K. Semi-parametric image inpainting[J]. arXiv:1807.02855, 2018. [85] RUSSAKOVSKY O, DENG J, SU H, et al. ImageNet large scale visual recognition challenge[J]. International Journal of Computer Vision, 2015, 115(3): 211-252. [86] ZHOU B, LAPEDRIZA A, KHOSLA A, et al. Places: a 10 million image database for scene recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(6): 1452-1464. [87] DOERSCH C, SINGH S, GUPTA A, et al. What makes paris look like paris?[J]. ACM Transactions on Graphics, 2012, 31(4): 101. [88] LIU Z, LUO P, WANG X, et al. Deep learning face attributes in the wild[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision, 2015: 3730-3738. [89] KARRAS T, AILA T, LAINE S, et al. Progressive growing of GANs for improved quality, stability, and variation[J]. arXiv:1710.10196, 2017. [90] KARRAS T, LAINE S, AILA T. A style-based generator architecture for generative adversarial networks[C]//Proce-edings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 4401-4410. |
[1] | WANG Cailing, YAN Jingjing, ZHANG Zhidong. Review on Human Action Recognition Methods Based on Multimodal Data [J]. Computer Engineering and Applications, 2024, 60(9): 1-18. |
[2] | LIAN Lu, TIAN Qichuan, TAN Run, ZHANG Xiaohang. Research Progress of Image Style Transfer Based on Neural Network [J]. Computer Engineering and Applications, 2024, 60(9): 30-47. |
[3] | YANG Chenxi, ZHUANG Xufei, CHEN Junnan, LI Heng. Review of Research on Bus Travel Trajectory Prediction Based on Deep Learning [J]. Computer Engineering and Applications, 2024, 60(9): 65-78. |
[4] | SONG Jianping, WANG Yi, SUN Kaiwei, LIU Qilie. Short Text Classification Combined with Hyperbolic Graph Attention Networks and Labels [J]. Computer Engineering and Applications, 2024, 60(9): 188-195. |
[5] | LI Houjun, WEI Boquan. Attribute Distillation for Zero-Shot Recognition [J]. Computer Engineering and Applications, 2024, 60(9): 219-227. |
[6] | CHE Yunlong, YUAN Liang, SUN Lihui. 3D Object Detection Based on Strong Semantic Key Point Sampling [J]. Computer Engineering and Applications, 2024, 60(9): 254-260. |
[7] | QIU Yunfei, WANG Yifan. Multi-Level 3D Point Cloud Completion with Dual-Branch Structure [J]. Computer Engineering and Applications, 2024, 60(9): 272-282. |
[8] | YE Bin, ZHU Xingshuai, YAO Kang, DING Shangshang, FU Weiwei. Binocular Depth Measurement Method for Desktop Interaction Scene [J]. Computer Engineering and Applications, 2024, 60(9): 283-291. |
[9] | ZHOU Bojun, CHEN Zhiyu. Survey of Few-Shot Image Classification Based on Deep Meta-Learning [J]. Computer Engineering and Applications, 2024, 60(8): 1-15. |
[10] | SUN Shilei, LI Ming, LIU Jing, MA Jingang, CHEN Tianzhen. Research Progress on Deep Learning in Field of Diabetic Retinopathy Classification [J]. Computer Engineering and Applications, 2024, 60(8): 16-30. |
[11] | WANG Weitai, WANG Xiaoqiang, LI Leixiao, TAO Yihao, LIN Hao. Review of Construction and Applications of Spatio-Temporal Graph Neural Network in Traffic Flow Prediction [J]. Computer Engineering and Applications, 2024, 60(8): 31-45. |
[12] | XIE Weiyu, ZHANG Qiang. Review on Detection of Drones and Birds in Photoelectric Images Based on Deep Learning Convolutional Neural Network [J]. Computer Engineering and Applications, 2024, 60(8): 46-55. |
[13] | SHEN Haiyun, HUANG Zhongyi, WANG Haichuan, YU Honghao. Improved Tracktor-Based Pedestrian Multi-Objective Tracking Algorithm [J]. Computer Engineering and Applications, 2024, 60(8): 242-249. |
[14] | ZHOU Dingwei, HU Jing, ZHANG Liangrui, DUAN Feiya. Collaborative Correction Technology of Label Omission in Dataset for Object Detection [J]. Computer Engineering and Applications, 2024, 60(8): 267-273. |
[15] | CHANG Xilong, LIANG Kun, LI Wentao. Review of Development of Deep Learning Optimizer [J]. Computer Engineering and Applications, 2024, 60(7): 1-12. |
Viewed | ||||||
Full text |
|
|||||
Abstract |
|
|||||