Computer Engineering and Applications ›› 2024, Vol. 60 ›› Issue (1): 217-226.DOI: 10.3778/j.issn.1002-8331.2207-0415

• Graphics and Image Processing • Previous Articles     Next Articles

Coarse and Fine Dual Path Image Inpainting Algorithm Based on Fourier Transform

CHEN Gang, SHENG Kuang, YANG Zhenguo, LIU Wenyin   

  1. 1.School of Computer, Guangdong University of Technology, Guangzhou 510006, China
    2.School of Artificial Intelligence, Guangdong Open University, Guangzhou 510091, China
    3.Cyberspace Security Research Center, Peng Cheng Laboratory, Shenzhen, Guangdong 518005, China
  • Online:2024-01-01 Published:2024-01-01



  1. 1.广东工业大学 计算机学院,广州 510006
    2.广东开放大学 人工智能学院,广州 510091
    3.鹏城实验室 网络空间安全研究中心,广东 深圳 518005

Abstract: In view of the weak ability of extracting global features and the frequency domain difference between the inpainted image and the original image in the traditional coarse and fine dual path image inpainting algorithm, causing the global structure of the repaired image to be poor, the coarse and fine dual path image inpainting algorithm based on Fourier transform is proposed. Firstly, in order to improve the ability of the encoder to extract features, an encoder based on squeeze and excitation model is designed to improve the ability of the network to collect global information. Secondly, the focal frequency loss is introduced for the first time to supervise image inpainting, and the frequency domain difference between the inpainted image and the original image is reduced, it improves the ability of the algorithm to inpaint high-frequency components, and artifacts and blurs are improved. Finally, the algorithm is applied to CelebA dataset. Compared with the images inpainted by the baseline algorithms, the peak signal-to-noise ratio (PNSR) and structural similarity (SSIM) of the images inpainted by the proposed algorithm are improved by 1.18%~6.14% and 0.11%~2.24% respectively. However, the distance score (FID) is decreased by 34.58%~38.78%. Experimental results show that the proposed algorithm achieves better performance improvement at a small time cost and enhances the global structure and clarity of the inpainted image.

Key words: squeeze and excitation model, Fourier transform, focal frequency loss, image inpainting

摘要: 针对传统的粗细双路径图像修复算法在修复图像时提取全局特征能力弱和所修复图像与原图像存在频域差,导致修复的图像全局结构差和存在伪影的问题,提出了傅里叶变换下的粗细双路径图像修复算法。为了改善编码器特征提取能力,设计了具有压缩奖惩机制的编码器来提升网络采集全局信息的能力;在编码器训练时首次引入焦频损失来监督图像的修复,缩小了修复图像与原图像的频域差,提升了算法修复高频成分的能力,改善了修复图像伪影和模糊性。将该算法应用于CelebA数据集,所提的算法修复的图像比基线算法所修复的图像的峰值信噪比(PNSR)、结构相似性(SSIM)等性能分别提高了1.18%~6.14%,0.11%~2.24%,而距离得分(FID)降低了34.58%~38.79%。实验结果表明,所提算法以微小的时间成本获取了较好的性能提升,增强了修复图像的全局结构性和清晰度。

关键词: 压缩奖惩块, 傅里叶变换, 焦频损失, 图像修复