隐空间转换的混合样本图像去雾

doi:10.3778/j.issn.1002-8331.2201-0105

摘要/Abstract

摘要： 深度学习从数据集中学习样本的内在规律，数据集的质量一定程度上决定了模型的表现。在去雾任务的公开数据集中，由于缺少成对真实数据，合成的成对数据难以模拟真实环境等问题，可能导致训练出的模型在实际环境中表现不佳。为此，提出混合样本学习问题，利用合成的成对数据和真实数据（混合样本）同时训练模型,通过隐空间的转换实现混合样本间的转换。算法利用变分自编码器和生成对抗网络（VAE-GAN）将混合样本分别编码到隐空间，利用对抗损失将真实数据的隐空间向合成雾图的隐空间对齐，利用含特征自适应融合（MFF）模块的映射网络学习成对数据隐空间之间的转换，从而建立起从真实雾图域到清晰图像域之间的去雾数据通路。实验结果表明，该算法相比其他去雾算法在真实雾图上的去雾结果更加清晰，对于较厚的雾图也有突出的效果，且该算法的峰值信噪比高于对比算法。

关键词: 单幅图像去雾, 隐空间转换, 混合样本, 变分自编码器（VAE）, 生成对抗网络（GAN）

Abstract: Deep learning learns the inherent laws of samples from datasets which determine the performance of the model to a certain extent. However, it may be lack of paired real data, or difficult to synthesize paired data to simulate the real environment to train in single image dehazing dataset. This problem may cause the trained model doesn’t perform well in real hazy image. This paper proposes hybrid samples learning problem, and the hybrid samples learning algorithm based on latent space translation, aiming to make full use of paired data and unpaired data（hybrid samples）. VAE-GAN（variational auto-encoder, generative adversarial networks） is used to encode hybrid samples into latent space, and then the adversarial loss is used to align real data with synthesis data. The mixup of feature adaptive fusion（MFF） module included in mapping net is used to learn the translation between paired data. So that, a dehazing data path from the real hazy image to the clear image is established. The experimental results show that proposed model performs well in real hazy images compared with other algorithms, and has outstanding effect on thick hazy images, and the peak signal to noise ratio of the proposed algorithm is higher than that of comparison algorithms.

Key words: single image dehazing, latent space translation, hybrid sample, variational autoencoder（VAE）, generative adversarial network（GAN）

郑玉彤, 孙昊英, 宋伟. 隐空间转换的混合样本图像去雾[J]. 计算机工程与应用, 2023, 59(9): 225-236.

ZHENG Yutong, SUN Haoying, SONG Wei. Hybrid Samples Image Dehazing via Latent Space Translation[J]. Computer Engineering and Applications, 2023, 59(9): 225-236.

参考文献

[1] HE K M，SUN J，TANG X O.Single image haze removal using dark channel prior[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2010，33（12）：2341-2353.
[2] 郭珈，王孝通，胡程鹏，等.基于单幅图像景深和大气散射模型的去雾方法[J].中国图象图形学报，2012，17（1）：27-32.
GUO J，WANG X T，HU C P，et al.Single image dehazing based on scene depth and physical model[J].Journal of Image and Graphics，2012，17（1）：27-32.
[3] FATTAL R.Single image dehazing[J].ACM Transactions on Graphics，2008，27（3）：1-9.
[4] ZHU Q S，MAI J M，SHAO L.A fast single image haze removal algorithm using color attenuation prior[J].IEEE Transactions on Image Processing，2015，24（11）：3522-3533.
[5] 胡伟，袁国栋，董朝，等.基于暗通道优先的单幅图像去雾新方法[J].计算机研究与发展，2010，47（12）：2132-2140.
HU W，YUAN G D，DONG Z，et al.Improved single image dehazing using dark channel prior[J].Journal of Computer Research and Development，2010，47（12）：2132-2140.
[6] TAN R T.Visibility in bad weather from a single image[C]//2008 IEEE Conference on Computer Vision and Pattern Recognition，Anchorage，Jun 24-26，2008.Washington：IEEE Computer Society，2008：1-8.
[7] CAI B L，XU X M，JIA K，et al.Dehazenet：an end-to-end system for single image haze removal[J].IEEE Transactions on Image Processing，2016，25（11）：5187-5198.
[8] LI B Y，PENG X L，WANG Z Y，et al.Aod-net：all-in-one dehazing network[C]//Proceedings of the IEEE International Conference on Computer Vision，Venice，Oct 22-29，2017.Washington：IEEE Computer Society，2017：4770-4778.
[9] REN W Q，LIU S，ZHANG H，et al.Single image dehazing via multi-scale convolutional neural networks[C]//European Conference on Computer Vision，Ansterdam，Oct 8-10.Berlin，Heidelberg：Springer，2016：154-169.
[10] 麦嘉铭，王美华，梁云，等.特征学习的单幅图像去雾算法[J].中国图象图形学报，2016，21（4）：464-474.
MAI J M，WANG M H，LIANG Y，et al.Single image dehazing algorithm by feature learning[J].Journal of Image and Graphics，2016，21（4）：464-474.
[11] QIN X，WANG Z L，BAI Y C，et al.FFA-Net：feature fusion attention network for single image dehazing[C]//Proceedings of the AAAI Conference on Artificial Intelligence，New York，Feb 7-12，2020.Menlo Park，CA：AAAI，2020：11908-11915.
[12] LIU X，MA Y，SHI Z，et al.Griddehazenet：attention-based multi-scale network for image dehazing[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，Seoul，Oct 27-Nov 2，2019.Piscataway：IEEE，2019：7314-7323.
[13] 杨爱萍，刘瑾，邢金娜，等.基于内容特征和风格特征融合的单幅图像去雾网络[J].自动化学报，2023，49（4）：1-9.
YANG A P，LIU J，XING J N，et al.Content feature and style feature fusion network for single image dehazing[J].Acta automatica Sinica，2023，49（4）：1-9.
[14] 寇大磊，钱敏，权冀川，等.基于多尺度卷积网络的快速图像去雾算法[J].计算机工程与应用，2020，56（20）：191-198.
KOU D L，QIAN M，QUAN J，et al.Fast image dehazing algorithm based on multi-scale convolutional network[J].Computer Engineering and Applications，2020，56（20）：191-198.
[15] 王硕，陈金玉.自适应校正透射率的暗通道先验去雾算法[J].计算机工程与应用，2021，57（13）：207-211.
WANG S，CHEN J Y.Dark channel prior defogging algorithm for adaptive correction transmittance[J].Computer Engineering and Applications，2021，57（13）：207-211.
[16] 刘兴瑞.自适应直方图均衡处理下船舶视频监控图像去雾算法[J].舰船科学技术，2020，42（16）：74-76.
LIU X R.Defogging algorithm of ship video surveillance image based on adaptive histogram equalization[J].Ship Science and Technology，2020，42（16）：74-76.
[17] ENGIN D，GEN A，EKENEL H K.Cycle-dehaze：enhanced cycleGAN for single image dehazing[C]//Proceedings of the 2018 IEEE on Conference on Computer Vision and Pattern Recognition，Salt Lake City，June 18-22，2018.Washington：IEEE Computer Society，2018：937-938.
[18] QU Y Y，CHEN Y Z，HUANG J Y，et al.Enhanced pix2pix dehazing network[C]//Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition，Long Beach，Jun 16-20，2019.Piscataway：IEEE Computer Society，2019：8152-8160.
[19] LI R D，PAN J S，LI Z C，et al.Single image dehazing via conditional generative adversarial network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Salt Lake City，Jun 18-22，2018.Washington：IEEE Computer Society，2018：8202-8211.
[20] ISOLA P，ZHU J Y，ZHOU T H，et al.Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Honolulu，July 21-26，2017.Washington：IEEE Computer Society，2017：1125-1134.
[21] WANG T C，LIU M Y，ZHU J Y，et al.High-resolution image synthesis and semantic manipulation with conditional GANs[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Salt Lake City，June 18-23，2018.Washington：IEEE Computer Society，2018：8798-8807.
[22] LIU M Y，BREUEL T，KAUTZ J.Unsupervised image-to-image translation networks[C]//Advances in Neural Information Processing Systems，Long Beach California，Dec 4-9，2017.Cambridge：MIT Press，2017：700-708.
[23] ZHU J Y，PARK T，ISOLA P，et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision，Venice，Oct 22-29，2017.Washington：IEEE Computer Society，2017：2223-2232.
[24] KINGMA D P，WELLING M.Auto-encoding variational bayes[EB/OL].（2013-12-20）[2021-10-26].https：//arxiv.org/pdf/1312.6114.pdf.
[25] KIM T K，PAIK J K，KANG B S.Contrast enhancement system using spatially adaptive histogram equalization with temporal filtering[J].IEEE Transactions on Consumer Electronics，1998，44（1）：82-87.
[26] NARASIMHAN S G，NAYAR S K.Chromatic framework for vision in bad weather[C]//Proceedings IEEE Conference on Computer Vision and Pattern Recognition，Hilton Head，June 15，2000.Washington：IEEE Computer Society，2000：598-605.
[27] NASASIMHAN S G，NAYAR S K.Vision and the atmosphere[J].International Journal of Computer Vision，2002，48（3）：233-254.
[28] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial networks[J].Communications of the ACM，2020，63（11）：139-144.
[29] RONNEBERGER O，FISCHER P，BROX T.U-net：convolutional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention，Heidelberg，March，12-14，2017.Berlin，Heidelberg：Springer，2017：234-241.
[30] MIRZA M，OSINDERO S.Conditional generative adversarial nets[EB/OL].（2014-11-06）[2021-10-26].https：//arxiv.org/pdf/1411.1784.pdf.
[31] DONG Y，LIU Y H，ZHANG H，et al.FD-GAN：generative adversarial networks with fusion-discriminator for single image dehazing[C]//Proceedings of the AAAI Conference Artificial Intelligence，New York，Feb 7-12，2020.Menlo Park，CA：AAAI，2020：10729-10736.
[32] JOHNSON J，ALAHI A，LI F F.Perceptual losses for real-time style transfer and super-resolution[C]//European Conference on Computer Vision，Amsterdam，Oct 11-14，2016.Berlin，Heidelberg：Springer，2016：694-711.
[33] ZHANG R，ISOLA P，EFROS A A，et al.The unreasonable effectiveness of deep features as a perceptual metric[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Salt Lake City，June 18-23，2018.Washington：IEEE Computer Society，2018：586-595.
[34] LI B Y，REN W Q，FU D P，et al.Benchmarking single image dehazing and beyond[J].IEEE Transactions on Image Processing，2018，28（1）：492-505.
[35] ZHANG Y F，LO D，SHARMA G.HazeRD：an outdoor scene dataset and benchmark for single image dehazing[C]//2017 IEEE International Conference on Image Processing（ICIP），Beijing，Sept 17-20，2017.Washington：IEEE Computer Society，2017：3205-3209.
[36] WAN Z Y，ZHANG B，CHEN D D，et al.Bringing old photos back to life[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，Seattle，June 13-19，2020.Washington：IEEE Computer Society，2020：2747-2757.
[37] BEN-DAVID S，BLITZER J，CRAMMER K，et al.Analysis of representations for domain adaptation[J].Advances in Neural Information Processing Systems，2007，19：137.
[38] MAO X D，LI Q，XIE H R，et al.Least squares generative adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision，Venice，Oct 22-29，2017.Washington：IEEE Computer Society，2017：2794-2802.
[39] WU H Y，QU Y Y，LIN S H，et al.Contrastive learning for compact single image dehazing[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，Nashville，June 20-25，2021.Piscataway，NJ：IEEE，2021：10551-10560.
[40] ZHANG H，CISSE M，DAUPHIN Y N，et al.Mixup：beyond empirical risk minimization[EB/OL].（2017-10-25）[2021-10-26].https：//arxiv.org/pdf/1710.09412.pdf.
[41] SILBERMAN N，HOIEM D，KOHLI P，et al.Indoor segmentation and support inference from RGBD images[C]//European Conference on Computer Vision，Firenze，Oct 7-13，2012.Berlin，Heidelberg：Springer，2012：746-760.
[42] ANCUTI C O，ANCUTI C，TIMOFTE R.NH-HAZE：an image dehazing benchmark with non-homogeneous hazy and haze-free images[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，Seattle，June 14-19，2020.Piscataway，NJ：IEEE，2020：444-445.
[43] ANCUTI C O，ANCUTI C，SBERT M，et al.Dense-haze：a benchmark for image dehazing with dense-haze and haze-free images[C]//Proceedings of the IEEE International Conference on Image Processing，Taipei，China，Sept 22-25，2019.Piscataway，NJ：IEEE，2019：1014-1018.
[44] ZHENG Z，REN W，CAO X，et al.Ultra-high-definition image dehazing via multi-guided bilateral learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，June 21-24，2021.Piscataway，NJ：IEEE，2021：16180-16189.