Coarse and Fine Dual Path Image Inpainting Algorithm Based on Fourier Transform

doi:10.3778/j.issn.1002-8331.2207-0415

Abstract

Abstract: In view of the weak ability of extracting global features and the frequency domain difference between the inpainted image and the original image in the traditional coarse and fine dual path image inpainting algorithm, causing the global structure of the repaired image to be poor, the coarse and fine dual path image inpainting algorithm based on Fourier transform is proposed. Firstly, in order to improve the ability of the encoder to extract features, an encoder based on squeeze and excitation model is designed to improve the ability of the network to collect global information. Secondly, the focal frequency loss is introduced for the first time to supervise image inpainting, and the frequency domain difference between the inpainted image and the original image is reduced, it improves the ability of the algorithm to inpaint high-frequency components, and artifacts and blurs are improved. Finally, the algorithm is applied to CelebA dataset. Compared with the images inpainted by the baseline algorithms, the peak signal-to-noise ratio (PNSR) and structural similarity (SSIM) of the images inpainted by the proposed algorithm are improved by 1.18%~6.14% and 0.11%~2.24% respectively. However, the distance score (FID) is decreased by 34.58%~38.78%. Experimental results show that the proposed algorithm achieves better performance improvement at a small time cost and enhances the global structure and clarity of the inpainted image.

Key words: squeeze and excitation model, Fourier transform, focal frequency loss, image inpainting

摘要： 针对传统的粗细双路径图像修复算法在修复图像时提取全局特征能力弱和所修复图像与原图像存在频域差，导致修复的图像全局结构差和存在伪影的问题，提出了傅里叶变换下的粗细双路径图像修复算法。为了改善编码器特征提取能力，设计了具有压缩奖惩机制的编码器来提升网络采集全局信息的能力；在编码器训练时首次引入焦频损失来监督图像的修复，缩小了修复图像与原图像的频域差，提升了算法修复高频成分的能力，改善了修复图像伪影和模糊性。将该算法应用于CelebA数据集，所提的算法修复的图像比基线算法所修复的图像的峰值信噪比（PNSR）、结构相似性（SSIM）等性能分别提高了1.18%~6.14%，0.11%~2.24%，而距离得分（FID）降低了34.58%~38.79%。实验结果表明，所提算法以微小的时间成本获取了较好的性能提升，增强了修复图像的全局结构性和清晰度。

关键词: 压缩奖惩块, 傅里叶变换, 焦频损失, 图像修复

CHEN Gang, SHENG Kuang, YANG Zhenguo, LIU Wenyin. Coarse and Fine Dual Path Image Inpainting Algorithm Based on Fourier Transform[J]. Computer Engineering and Applications, 2024, 60(1): 217-226.

陈刚, 盛况, 杨振国, 刘文印. 傅里叶变换下的粗细双路径图像修复算法[J]. 计算机工程与应用, 2024, 60(1): 217-226.

References

[1] BERTALMIO M, SAPIRO G, CASELLES V, et al. Image inpainting[C]//Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, 2000: 417-424.
[2] LIU Y, SHU C. A comparison of image inpainting techniques[C]//Sixth International Conference on Graphic and Image Processing, 2015: 347-357.
[3] CLERCQ R D. The metaphysics of art restoration[J]. British Journal of Aesthetics, 2013, 53(3): 261-275.
[4] ZHANG Z. Research on the taxi traffic accident and violation identification model[C]//2010 2nd International Asia Conference on Informatics in Control, Automation and Robotics, 2010: 533-536.
[5] WANG C, DONG E, YANG S. Occlusion face image inpainting method based on multi-scale and contextual attention[C]//2021 China Automation Congress, 2021: 2857-2861.
[6] MARTY P F. Museum websites and museum visitors: digital museum resources and their use[J]. Museum Management and Curatorship, 2008, 23(1): 81-99.
[7] PATHAK D, KRAHENBUHL P, DONAHUE J, et al. Context encoders: feature learning by inpainting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 2536-2544.
[8] JIANG Y, XU J, YANG B, et al. Image inpainting based on generative adversarial networks[J]. IEEE Access, 2020, 8: 22884-22892.
[9] SAGONG M, SHIN Y, KIM S, et al. Pepsi: fast image inpainting with parallel decoding network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 11360-11368.
[10] 路志英, 周庆霞, 李鑫. 基于改进Criminisi算法的地基云图修复方法[J]. 数据采集与处理, 2019, 34(1): 12-21.
LU Z Y, ZHOU Q X, LI X. Ground-based cloud image inpainting method based on improved Criminisi algorithm[J]. Journal of Data Acquisition and Processing, 2019, 34(1): 12-21.
[11] ENGAN K, AASE S O, HUSOY J H. Method of optimal directions for frame design[C]//IEEE International Conference on Acoustics, Speech, and Signal Processing, 1999: 2443-2446.
[12] WANG Y, TAO X, QI X, et al. Image inpainting via generative multi-column convolutional neural networks[C]//Proceedings of the 2018 Conference and Workshop on Neural Information Processing Systems, 2018: 329-338.
[13] ZENG Y, FU J, CHAO H, et al. Learning pyramid-context encoder network for high-quality image inpainting[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 1486-1494.
[14] ZHENG C, CHAMT J, CAI J. Pluralistic image completion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 1438-1447.
[15] YU J, LIN Z, YANG J, et al. Generative image inpainting with contextual attention[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 5505-5514.
[16] 罗三定, 涂宙霖, 田光亚. 基于频率谱处理的瓷砖纹理分类方法[J]. 计算机工程与应用, 2017, 53(4): 184-188.
LUO S D, TU Z L, TIAN G Y. Ceramic tiles texture classification method based on frequency spectrum processing[J]. Computer Engineering and Applications, 2017, 53(4): 184-188.
[17] 王晋宇, 杨海涛, 李高源, 等. 生成对抗网络及其图像处理应用研究进展[J]. 计算机工程与应用, 2021, 57(8): 26-35.
WANG J Y, YANG H T, LI G Y, et a al. Research progess of generative adversarial network and its application in image processing[J]. Computer Engineering and Applications, 2021, 57(8): 26-35.
[18] 董张慧雅, 张凡, 王莉. 基于联合注意力生成对抗网络的自动文摘模型[J]. 计算机工程与设计, 2021, 42(6): 1756-1762.
DONG Z H Y, ZHANG F, WANG L. Automatic summarization model based on joint attention generative adversarial network[J]. Computer Engineering and Design, 2021, 42(6): 1756-1762.
[19] JIANG L, DAI B, WU W, et al. Focal frequency loss for image reconstruction and synthesis[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 13919-13929.
[20] HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 7132-7141.
[21] XU Z Q J, ZHAGN Y, LUO T, et al. Frequency principle: Fourier analysis sheds light on deep neural networks[J]. arXiv:1901.06523, 2019.
[22] XU Z Q J, ZHANG Y, LUO T. Overview frequency principle/spectral bias in deep learning[J]. arXiv:2201.07395, 2022.
[23] CHU J, GUO Z, LENG L. Object detection based on multi-layer convolution feature fusion and online hard example mining[J]. IEEE Access, 2018, 6: 19959-19967.
[24] MUKHOTI J, KULHARIA V, SANYAL A, et al. Calibrating deep neural networks using focal loss[C]//Advances in Neural Information Processing Systems, 2020: 15288-15299.
[25] CHUA L O. CNN: a vision of complexity[J]. International Journal of Bifurcation and Chaos, 1997, 7(10): 2219-2425.
[26] YU J, LIU Z, YANG J, et al. Free-form image inpainting with gated convolution[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019: 4471-4480.
[27] PECHA M, HORAK D. Analyzing l1-loss and l2-loss support vector machines implemented in PERMON toolbox[C]//International Conference on Advanced Engineering Theory and Applications. Cham: Springer, 2018: 13-23.
[28] 李健, 孙大松, 张备伟. 结合双编码器与对抗训练的图像修复[J]. 计算机工程与应用, 2021, 57(7): 192-197.
LI J, SUN D S, ZHANG B W. Image restoration using dual-encoder and adversarial training[J]. Computer Engineering and Applications, 2021, 57(7): 192-197.