雨痕退化预测与预训练扩散先验的单图像去雨方法

doi:10.3778/j.issn.1002-8331.2311-0264

摘要/Abstract

摘要： 雨天拍摄的图像由于雨痕干扰，会降低视觉质量和后续任务精度。为有效应用扩散模型中的生成式先验以及避免重新训练条件扩散模型带来的计算负担，提出了一种结合雨痕退化预测与无条件预训练扩散模型的单图像去雨方法，通过使用卷积字典学习机制在雨痕退化预测网络中获取带雨图像的雨痕图，将雨痕图用于引导零空间扩散模型，实现了使用在已有的预训练无条件扩散模型下进行单图像去雨，从而有效地提高了图像去雨的质量。和其他单图像去雨方法相比，该方法在Rain100H和Rain100L数据集上取得了目前最好的结果，PSNR指标最大提升了0.44?dB（+1.1%），SSIM指标最大提升了0.006（+0.7%），LPIPS指标最大提升了0.008（+42.1%）。

关键词: 单图像去雨, 扩散模型, 预训练模型

Abstract: Images captured in rainy weather degrade visual quality and subsequent task accuracy due to interference from rain streaks. To effectively apply the generative prior in the diffusion model as well as to avoid the computational burden of re-training the conditional diffusion model, a single image deraining method combining rain streak degradation prediction and unconditional pre-trained diffusion model is proposed, which is achieved by using the convolutional dictionary learning mechanism to obtain rain streak maps with raining images, the rain streak map will be used to bootstrap the null-space diffusion model. Single image deraining using an existing pre-trained unconditional diffusion model, thus effectively improving the quality of image deraining. Compared with other single image deraining methods, the method achieves the highest PSNR improvement of 0.44?dB (+1.1%), SSIM improvement of 0.006 (+0.7%), and LPIPS improvement of 0.008 (+42.1%) on the Rain100H and Rain100L datasets, achieving the state-of-the-art results so far on both datasets.

Key words: single image deraining, diffusion model, pre-trained model

谢瑞麟, 吴昊, 袁国武. 雨痕退化预测与预训练扩散先验的单图像去雨方法[J]. 计算机工程与应用, 2025, 61(6): 304-316.

XIE Ruilin, WU Hao, YUAN Guowu. Single Image Deraining Using Rainy Streak Degradation Prediction and Pre-Trained Diffusion Prior[J]. Computer Engineering and Applications, 2025, 61(6): 304-316.

参考文献

[1] WANG H, WU Y, LI M, et?al. A survey on rain removal from video and single image[J]. Science China Information Sciences, 2022, 65: 111101.
[2] ANWAR S, TAHIR M, LI C, et?al. Image colorization: a survey and dataset[J]. arXiv:2008.10774, 2020.
[3] LARSSON G, MAIRE M, SHAKHNAROVICH G. Learning representations for automatic colorization[C]//Proceedings of the European Conference on Computer Vision (ECCV 2016), 2016: 577-593.
[4] KAWAR B, ELAD M, ERMON S, et?al. Denoising diffusion restoration models[C]//Advances in Neural Information Processing Systems, 2022: 23593-23606.
[5] WANG Y, YU J, ZHANG J. Zero-shot image restoration using denoising diffusion null-space model[C]//Proceedings of the Eleventh International Conference on Learning Representations, 2023.
[6] LEDIG C, THEIS L, HUSZAR F, et?al. Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[7] HARIS M, SHAKHNAROVICH G, UKITA N. Deep back-projection networks for super-resolution[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[8] KANG L W, LIN C W, FU Y H. Automatic single-image-based rain streaks removal via image decomposition[J]. IEEE Transactions on Image Processing, 2012, 21(4): 1742-1755.
[9] ZHANG H, PATEL V?M. Convolutional sparse and low-rank coding-based rain streak removal[C]//Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), 2017.
[10] LI Y, TAN R?T, GUO X, et?al. Rain streak removal using layer priors[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[11] YANG W, TAN R?T, FENG J, et?al. Deep joint rain detection and removal from a single image[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[12] ZHANG H, SINDAGI V, PATEL V?M. Image de-raining using a conditional generative adversarial network[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(11): 3943-3956.
[13] GOU Y, LI B, LIU Z, et?al. CLEARER: multi-scale neural architecture search for image restoration[C]//Advances in Neural Information Processing Systems, 2020.
[14] WANG H, YUE Z, XIE Q, et?al. From rain generation to rain removal[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[15] SHAO M, LI L, WANG H, et?al. Selective generative adversarial network for raindrop removal from a single image[J]. Neurocomputing, 2021, 426: 265-273.
[16] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et?al. Generative adversarial networks[J]. Communications of the ACM, 2020, 63(11): 139-144.
[17] HO J, JAIN A, ABBEEL P. Denoising diffusion probabilistic models[C]//Advances in Neural Information Processing Systems, 2020: 6840-6851.
[18] JABBAR A, LI X, OMAR B. A survey on generative adversarial networks: variants, applications, and training[J]. ACM Computing Surveys, 2022, 54(8): 1-49.
[19] NICHOL A, DHARIWAL P, RAMESH A, et?al. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models[J]. arXiv:2112.10741, 2021.
[20] NICHOL A?Q, DHARIWAL P. Improved denoising diffusion probabilistic models[C]//Proceedings of the International Conference on Machine Learning, 2021: 8162-8171.
[21] RAMESH A, DHARIWAL P, NICHOL A, et?al. Hierarchical text-conditional image generation with CLIP latents[J]. arXiv: 2204.06125, 2022.
[22] CROITORU F A, HONDRU V, IONESCU R?T, et?al. Diffusion models in vision: a survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(9): 10850-10869.
[23] ZHU Y, ZHANG K, LIANG J, et?al. Denoising diffusion models for plug-and-play image restoration[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (NTIRE), 2023.
[24] FEI B, LYU Z, PAN L, et?al. Generative diffusion prior for unified image restoration and enhancement[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 9935-9946.
[25] DENG J, DONG W, SOCHER R, et?al. ImageNet: a large-scale hierarchical image database[C]//Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009.
[26] DHARIWAL P, NICHOL A. Diffusion models beat GANs on image synthesis[C]//Advances in Neural Information Processing Systems, 2021: 8780-8794.
[27] HUYNH-THU Q, GHANBARI M. Scope of validity of PSNR in image/video quality assessment[J]. Electronics Letters, 2008, 44(13): 800.
[28] WANG Z, BOVIK A, SHEIKH H, et?al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[29] ZHANG R, ISOLA P, EFROS A?A, et?al. The unreasonable effectiveness of deep features as a perceptual metric[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[30] EIGEN D, KRISHNAN D, FERGUS R. Restoring an image taken through a window covered with dirt or rain[C]//Proceedings of the 2013 IEEE International Conference on Computer Vision, 2013.
[31] YU L, WANG B, HE J, et?al. Single image deraining with continuous rain density estimation[J]. IEEE Transactions on Multimedia, 2023, 25: 443-456.
[32] YADAV S, MEHRA A, ROHMETRA H, et?al. DerainGAN: single image deraining using Wasserstein GAN[J]. Multimedia Tools and Applications, 2021, 80(30): 36491-36507.
[33] WEI Y, ZHANG Z, WANG Y, et?al. Semi-deraingan: a new semi-supervised single image deraining[C]//Proceedings of the 2021 IEEE International Conference on Multimedia and Expo (ICME), 2021.
[34] WEI Y, ZHANG Z, ZHANG H, et?al. A coarse-to-fine multi-stream hybrid deraining network for single image deraining[C]//Proceedings of the 2019 IEEE International Conference on Data Mining (ICDM), 2019.
[35] LI B, LIU X, HU P, et?al. All-in-one image restoration for unknown corruption[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022: 17431-17441.
[36] YANG W, TAN R?T, FENG J, et?al. Joint rain detection and removal from a single image with contextualized deep networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(6): 1377-1393.
[37] FU X, HUANG J, DING X, et?al. Clearing the skies: a deep network architecture for single-image rain removal[J]. IEEE Transactions on Image Processing, 2017, 26(6): 2944-2956.
[38] XIA H, ZHUGE R, LI H, et?al. Single image rain removal via a simplified residual dense network[J]. IEEE Access, 2018, 6: 66522-66535.
[39] ZHANG H, PATEL V?M. Density-aware single image de-raining using a multi-stream dense network[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[40] REN D, ZUO W, HU Q, et?al. Progressive image deraining networks: a better and simpler baseline[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[41] WANG T, YANG X, XU K, et?al. Spatial attentive single-image deraining with a high quality real rain dataset[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[42] FU X, LIANG B, HUANG Y, et?al. Lightweight pyramid networks for image deraining[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(6): 1794-1807.
[43] YASARLA R, SINDAGI V?A, PATEL V?M. Syn2Real transfer learning for image deraining using Gaussian processes[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[44] VASWANI A, SHAZEER N, PARMAR N, et?al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017.
[45] KHAN S, NASEER M, HAYAT M, et?al. Transformers in vision: a survey[J]. ACM Computing Surveys, 2022, 54(10): 1-41.
[46] WANG Z, CUN X, BAO J, et?al. Uformer: a general U-shaped transformer for image restoration[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[47] TU Z, TALEBI H, ZHANG H, et?al. MAXIM: multi-axis MLP for image processing[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[48] CHEN X, WANG X, ZHOU J, et?al. Activating more pixels in image super-resolution transformer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 22367-22377.
[49] LIANG J, CAO J, SUN G, et?al. SwinIR: image restoration using swin transformer[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2021.
[50] CHEN Z, ZHANG Y, GU J, et?al. Cross aggregation transformer for image restoration[C]//Advances in Neural Information Processing Systems, 2022: 25478-25490.
[51] LIANG Y, ANWAR S, LIU Y. DRT: a lightweight single image deraining recursive transformer[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2022.
[52] CHEN X, LI H, LI M, et?al. Learning a sparse transformer network for effective image deraining[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023: 5896-5905.
[53] XIAO J, FU X, LIU A, et?al. Image de-raining transformer[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(11): 1-18.
[54] ZAMIR S?W, ARORA A, KHAN S, et?al. Restormer: efficient transformer for high-resolution image restoration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 5728-5739.
[55] CHEN H, WANG Y, GUO T, et?al. Pre-trained image processing transformer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 12299-12310.
[56] LIU Z, LIN Y, CAO Y, et?al. Swin Transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[57] HU X, FU C W, ZHU L, et?al. Depth-attentional features for single-image rain removal[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[58] QIAN R, TAN R?T, YANG W, et?al. Attentive generative adversarial network for raindrop removal from a single image[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[59] WEI Y, ZHANG Z, WANG Y, et?al. DerainCycleGAN: rain attentive CycleGAN for single image deraining and rainmaking[J]. IEEE Transactions on Image Processing, 2021, 30: 4788-4801.
[60] ZHU J Y, PARK T, ISOLA P, et?al. Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), 2017.
[61] SONG Y, ERMON S. Generative modeling by estimating gradients of the data distribution[C]//Advances in Neural Information Processing Systems, 2019.
[62] SOHL-DICKSTEIN J, WEISS E, MAHESWARANATHAN N, et?al. Deep unsupervised learning using nonequilibrium thermodynamics[C]//Proceedings of the International Conference on Machine Learning, 2015: 2256-2265.
[63] SONG J, MENG C, ERMON S. Denoising diffusion implicit models[C]//Proceedings of the International Conference on Learning Representations, 2020.
[64] SAHARIA C, CHAN W, SAXENA S, et?al. Photorealistic text-to-image diffusion models with deep language understanding[C]//Advances in Neural Information Processing Systems, 2022: 36479-36494.
[65] ROMBACH R, BLATTMANN A, LORENZ D, et?al. High-resolution image synthesis with latent diffusion models[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[66] LUO Z, GUSTAFSSON F?K, ZHAO Z, et?al. Image restoration with mean-reverting stochastic differential equations[C]//Proceedings of the International Conference on Machine Learning, 2023.
[67] SAHARIA C, HO J, CHAN W, et?al. Image super-resolution via iterative refinement[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(4): 4713-4726.
[68] XIA B, ZHANG Y, WANG S, et?al. DiffIR: efficient diffusion model for image restoration[J]. arXiv:2303.09472, 2023.
[69] SAHARIA C, CHAN W, CHANG H, et?al. Palette: image-to-image diffusion models[C]//Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings, 2022.
[70] LUGMAYR A, DANELLJAN M, ROMERO A, et?al. RePaint: inpainting using denoising diffusion probabilistic models[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[71] ?ZDENIZCI O, LEGENSTEIN R. Restoring vision in adverse weather conditions with patch-based denoising diffusion models[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(8): 10346-10357.
[72] WANG H, XIE Q, ZHAO Q, et?al. RCDNet: an interpretable rain convolutional dictionary network for single image deraining[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 35(6): 8668-8682.
[73] WANG H, XIE Q, ZHAO Q, et?al. A model-driven deep neural network for single image rain removal[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[74] MARTIN D, FOWLKES C, TAL D, et?al. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[C]//Proceedings Eighth IEEE International Conference on Computer Vision (ICCV 2001), 2002.
[75] KINGMA D, BA J. Adam: a method for stochastic optimization[J]. arXiv:1412.6980, 2014.
[76] ZAMIR S?W, ARORA A, KHAN S, et?al. Multi-stage progressive image restoration[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[77] LI X, WU J, LIN Z, et?al. Recurrent squeeze-and-excitation context aggregation net for single image deraining[C]//Proceedings of the European Conference on Computer Vision, 2018: 262-277.
[78] YASARLA R, PATEL V?M. Uncertainty guided multi-scale residual learning-using a cycle spinning CNN for single image de-raining[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[79] WEI W, MENG D, ZHAO Q, et?al. Semi-supervised transfer learning for image rain removal[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[80] PUROHIT K, SUIN M, RAJAGOPALAN A, et?al. Spatially-adaptive image restoration using distortion-guided networks[C]//Proceedings of the International Conference on Computer Vision, 2021.
[81] LI Y, LU J, CHEN H, et?al. Dilated convolutional transformer for high-quality image deraining[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2023: 4199-4207.
[82] FU X, HUANG J, ZENG D, et?al. Removing rain from single images via a deep detail network[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017: 3855-3863.
[83] LI S, REN W, ZHANG J, et?al. Single image rain removal via a deep decomposition-composition network[J]. Computer Vision and Image Understanding, 2019, 186: 48-57.

编辑推荐 0

Metrics

阅读次数

全文

HTML			PDF

最新录用	在线预览	正式出版	最新录用	在线预览	正式出版
0	0	0	0	0	39

	来源	本网站

	次数	39
	比例	100%

摘要

最新录用	在线预览	正式出版

0	0	13

	来源	本网站

	次数	14
	比例	100%