Perceptual Supervision-Guided and Multi-Hierarchical Feature Fusion for Image Dehazing

doi:10.3778/j.issn.1002-8331.2206-0024

Abstract

Abstract: Existing image dehazing methods do not consider whether the dehazed image meets human visual perception in the training of the network. With the encoder-decoder as the main structure, it is inevitable to lose detailed information. The image after dehazing has problems such as texture blur and color distortion. This paper aims at the above problems to propose a perceptual supervision-guided and multi-hierarchical feature fusion network for image dehazing. Firstly, in the network structure design, multi-hierarchical feature fusion modules are employed. In the encoder, the resolution-level feature reuse and fusion module is designed better to extract features with more robust representation at different scales and provide more detailed information for reconstructing high-quality images. In the feature transformation stage, the spatial context hierarchical feature extraction and fusion module is designed to extract and fuse the spatial context of different receptive fields to provide more accurate image structure information. In the decoder, an adaptive feature integration module is designed to adaptively fuse the features of different resolution levels generated in the down-sampling stage and the features of different spatial context levels generated in the feature conversion stage to improve the quality of image reconstruction. Secondly, perceptual loss and multiscale structural similarity loss are introduced in the loss function of the training phase to guide the network to learn more visual perceptual attributes. Compared with current mainstream methods, this method improves the visual effect on dehazed images while quantitative and qualitative metrics are significantly improved. Experimental results show significant dehazing effects on the RESIDE dataset and real hazy images.

Key words: image dehazing, perceptual supervision, encoder-decoder network, multi-hierarchical feature fusion

摘要： 现有图像去雾方法在网络训练时没有考虑去雾后的图像是否满足人类视觉感知；其次以编解码结构为主要结构的去雾网络，不可避免丢失细节信息，去雾后的图像存在纹理模糊、颜色失真等问题。针对上述问题，提出了一个基于感知监督和多层次特征融合的图像去雾网络。在网络结构中设计了不同层次的特征融合模块。在编码阶段设计分辨率层次特征复用与融合模块，更好地提取不同尺度下表达能力更强的特征，为重建高质量图像提供更多细节信息；特征转换阶段设计空间上下文层次特征提取与融合模块，提取与融合不同感受野的空间上下文的特征，以提供更加精准的图像结构信息；解码阶段设计自适应特征融合模块，自适应地融合下采样阶段生成的不同分辨率层次的特征及特征转换阶段输出的不同空间上下文层次的特征；其次在训练阶段的损失函数中引入感知损失和多尺度结构相似度损失，引导网络学习更多的视觉感知属性。与当前主流方法相比较，该方法在定量和定性指标得到明显提升的同时提高了对去雾图像的视觉效果。实验结果表明在RESIDE合成数据集以及真实有雾图像上取得显著的去雾效果。

关键词: 图像去雾, 感知监督, 编解码网络, 多层次特征融合

WU Junjiang, CHU Jun, LU Ang, LENG Lu. Perceptual Supervision-Guided and Multi-Hierarchical Feature Fusion for Image Dehazing[J]. Computer Engineering and Applications, 2023, 59(21): 204-213.

吴峻江, 储珺, 卢昂, 冷璐. 基于感知监督和多层次特征融合的去雾算法[J]. 计算机工程与应用, 2023, 59(21): 204-213.

References

[1] 谭显东，彭辉.改进YOLOv5的SAR图像舰船目标检测[J].计算机工程与应用，2022，58（4）：247-254.
TAN X D，PENG H.Improved YOLOv5 ship target detection in SAR image[J].Computer Engineering and Applications，2022，58（4）：247-254.
[2] 李彪，孙瑾，李星达，等.自适应特征融合的相关滤波跟踪算法[J].计算机工程与应用，2022，58（9）：208-218.
LI B，SUN J，LI X D，et al.Correlation filter target tracking based on adaptive multi-feature fusion[J].Computer Engineering and Applications，2022，58（9）：208-218.
[3] NARASIMHAN S G，NAYAR S K.Vision and the atmosphere[J].International Journal of Computer Vision，2002，48：233-254.
[4] HE K M，SUN J，TANG X O.Single image haze removal using dark channel prior[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2010，33（12）：2341-2353.
[5] ZHU Q，MAI J，SHAO L.A fast single image haze removal algorithm using color attenuation prior[J].IEEE Transactions on Image Processing，2015，24（11）：3522-3533.
[6] BERMAN D，AVIDAN S.Non-local image dehazing[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas，NV，USA：IEEE，2016：1674-1682.
[7] LI B，PENG X，WANG Z，et al.AOD-Net：all-in-one dehazing network[C]//2017 IEEE International Conference on Computer Vision.Venice，Italy：IEEE，2017：4770-4778.
[8] CAI B L，XU X M，JIA K，et al.DehazeNet：an end-to-end system for single image haze removal[J].IEEE Transactions on Image Processing，2016，25（11）：5187-5198.
[9] QU Y，CHEN Y，HUANG J，et al.Enhanced pix2pix dehazing network[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Long Beach，USA：IEEE，2019：8160-8168.
[10] DONG Y，LIU Y，ZHANG H，et al.FD-GAN：generative adversarial networks with fusion-discriminator for single image dehazing[C]//34th AAAI Conference on Artificial Intelligence.Hilton Midtown，New York：AAAI Press，2020：10729-10736.
[11] REN W，MA L，ZHANG J，et al.Gated fusion network for single image dehazing[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition.Salt-Lake City，UT，USA：IEEE，2018：3253-3261.
[12] WU H，LIU J，XIE Y，et al.Knowledge transfer dehazing network for non-homogeneous dehazing[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.Seattle，WA，USA：IEEE，2020：478-479.
[13] FENG T，WANG C，CHEN X，et al.URNet：a U-Net based residual network for image dehazing[J].Applied Soft Computing，2021，102（12）：106884.
[14] RONNEBERGER O，FISCHER P，BROX T.U-net：convolutional networks for biomedical image segmentation[C]//18th International Conference on Medical Image，Computing and Computer Assisted Intervention.Munich，Germany：Springer，2015：234-241.
[15] HE K M，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas，USA：IEEE，2016：770-778.
[16] MEI K，JIANG A，LI J，et al.Progressive feature fusion network for realistic image dehazing[C]//15th Asian Conference on Computer Vision.Perth，Australia：Springer，2018：203-215.
[17] LIU X，MA Y，SHI Z，et al.GridDehazeNet：attention-based multi-scale network for image dehazing[C]//2019 IEEE/CVF International Conference on Computer Vision，Long Beach，CA，USA，2019：7314-7323
[18] 杨爱萍，刘瑾，邢金娜，等.基于内容特征和风格特征融合的单幅图像去雾网络[J].自动化学报，2023，49（4）：769-777.
YANG A P，LIU J，XING J N，et al.Content feature and style feature fusion network for single image dehazing[J].Acta Automatica Sinica，2023，49（4）：769-777.
[19] DONG H，PAN J，XIANG L，et al.Multi-scale boosted dehazing network with dense feature fusion[C]//2020 IEEE Conference on Computer Vision and Pattern Recognition.Seattle，WA，USA：IEEE，2020：2157-2167.
[20] CHEN D D，HE M M，FAN Q N，et al.Gated context aggregation network for image dehazing and deraining[C]//2019 IEEE Winter Conference on Applications of Computer Vision.Hilton Waikoloa Village，Hawaii：IEEE，2019：1375-1383.
[21] YANG H H，YANG C H H，TSAI Y C J.Y-net：multi-scale feature aggregation network with wavelet structure similarity loss function for single image dehazing[C]//2020 IEEE International Conference on Acoustics，Speech and Signal Processing.Barcelona，Spain：IEEE，2020：2628-2632.
[22] ZHANG Y，TIAN Y，KONG Y，et al.Residual dense network for image restoration[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2020，43（7）：2480-2495.
[23] CHEN L，LU X，ZHANG J，et al.HINet：half instance normalization network for image restoration[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：182-192.
[24] QIN X，WANG Z L，BAI Y C，et al.FFA-Net：feature fusion attention network for single image dehazing[C]//34th AAAI Conference on Artificial Intelligence.Hilton Midtown，New York：AAAI Press，2020：11908-11915.
[25] WANG Z，JI S.Smoothed dilated convolutions for improved dense prediction[J].Data Mining and Knowledge Discovery，2021，35（4）：1470-1496.
[26] METWALY K，LI X，GUO T，et al.Non local channel attention for non-homogeneous image dehazing[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.Seattle，WA，USA：IEEE，2020：452-453
[27] YU M，CHERUKURI V，GUO T，et al.Ensemble dehazing networks for non-homogeneous haze[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.Seattle，WA，USA：IEEE，2020：450-451.
[28] LEDIG C，THEIS L，HUSZáR F，et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition.Honolulu，HI，USA：IEEE，2017：4681-4690.
[29] KUTTER M，WINKLER S.A vision-based masking model for spread spectrum image watermarking[J].IEEE Transactions on Image Processing，2002，11（1）：16-25.
[30] ZHAO H，GALLO O，FROSIO I，et al.Loss functions for image restoration with neural networks[J].IEEE Transactions on Computational Imaging，2016，3（1）：47-57.
[31] WATSON A B.DCT quantization matrices visually optimized for individual images[J].Human Vision Visual Processing & Digital Display IV，1993，1913：202-216.
[32] JOHNSON J，ALAHI A，LIU F F.Perceptual losses for real-time style transfer and super-resolution[C]//14th European Conference on Computer Vision.Munich，Germany：Springer，2016：694-711.
[33] RUSSAKOVSKY O，DENG J，SU H，et al.ImageNet large scale visual recognition challenge[J].International Journal of Computer Vision，2015，115（3）：211-252.
[34] LI B，REN W Q，FU D，et al.Benchmarking single image dehazing and beyond[J].IEEE Transactions on Image Processing，2018，28（1）：492-505.