Sand Dust Degradation Images Enhancement Algorithm via Multi-Branch Restoration Network

doi:10.3778/j.issn.1002-8331.2207-0214

Abstract

Abstract: At present, most of the dust image enhancement algorithms based on deep learning are similar to image defogging algorithms, but the effect is not ideal and weaker than some traditional image processing algorithms due to the following two reasons. Firstly, because the overall color of the sand dust images is yellow, the dehazing algorithm will ignore the color recovery of the sand dust images. Secondly, due to the lack of large-scale benchmark data set, it is very difficult for deep neural network to learn the mapping from sand dust images to clear images on limited data. Therefore, this paper proposes a sand dust degradation images enhancement algorithm based on multi-branch repair network. In addition, a new sand dust images dataset is constructed based on atmospheric scattering model. The algorithm divides the neural network into three subnets, including transfer learning subnet, color recovery subnet and data fitting subnet. Each subnet has its special role. The sand dust images are processed by three subnets, and then the three subnet results are mapped to a clear images through a learning fusion layer. The qualitative comparison in the experimental results shows that this method can effectively restore the details of the dust image and better restore the visual color of the image, and this method can produce a clearer image that is more in line with the human visual experience than other advanced methods. From the quantitative comparison, the PSNR and SSIM indexes of the proposed algorithm on the synthetic dataset are 0.783 and 0.012 higher than those of the advanced algorithms, respectively. The proposed algorithm achieves the best NIQE and PIQE indexes on the real image dataset.

Key words: sand dust images, sand dust images enhancement, branch network, sand dust image datasets, color correction, adaptive normalization

摘要： 目前基于深度学习的沙尘图像增强算法大多类似于图像去雾算法，但由于以下两个方面的原因效果并不理想且弱于一些传统图像处理算法：由于沙尘图像整体色彩偏黄，去雾算法会忽略沙尘图像的色彩恢复问题；由于缺乏大规模基准数据集，深度神经网络在有限的数据上学习从沙尘图像到清晰图像的映射是非常困难的。提出一种基于多分支修复网络的沙尘降质图像增强算法；此外，基于大气散射模型构建了一个新型沙尘图像数据集。算法将神经网络分为三个子网，包括迁移学习子网、色彩恢复子网和数据拟合子网，每个子网有其特殊的作用，沙尘图像分别经过三个子网处理，然后将三子网结果通过一个可学习的融合层映射为清晰图像。实验结果中定性比较表明该方法可以有效恢复沙尘图像细节，并较好恢复图像的视觉色彩，且该方法对比其他先进的方法可以产生更加符合人眼视觉体验的清晰图像；从定量比较中，在合成数据集上提出的算法相比于所对比的先进算法PSNR和SSIM指数分别提高了0.783和0.012，在真实图像数据集上提出的算法取得了最好的NIQE和PIQE指数。

关键词: 沙尘图像, 沙尘图像增强, 分支网络, 沙尘图像数据集, 颜色校正, 自适应归一化

DING Yuan, WU Kaijun. Sand Dust Degradation Images Enhancement Algorithm via Multi-Branch Restoration Network[J]. Computer Engineering and Applications, 2023, 59(24): 227-237.

丁元, 邬开俊. 多分支修复网络的沙尘降质图像增强算法[J]. 计算机工程与应用, 2023, 59(24): 227-237.

References

[1] CANTOR A.Optics of the atmosphere：scattering by molecules and particles[J].Optica Acta International Journal of Optics，1977，24（7）：779-779.
[2] HE K，SUN J，TANG X.Single images haze removal using dark channel prior[J].IEEE Transactions on Pattern Analysis and Machine Intellience，2011，33（12）：2341-2353.
[3] GAO G X，LAI H C，et al.Sand-dust image restoration based on reversing the blue channel prior[J].IEEE Photonics Journal，2020，12（2）：1-16.
[4] CAI B，XU X，JIA K，et al.DehazeNet：an end-to-end system for single image haze removal[J].IEEE Transactions on Image Processing，2016，25（11）：5187-5198.
[5] LI B，PENG X，WANG Z，et al.AOD-net：all-in-one dehazing network[C]//2017 IEEE International Conference on Computer Vision（ICCV），Venice，Italy，2017：4780-4788.
[6] QIN X，WANG Z，BAI Y，et al.FFA-Net：feature fusion attention network for single image dehazing[C]//National Conference on Artificial Intelligence，Association for the Advancement of Artificial Intelligence（AAAI），2020：11908-11915.
[7] DUDHANE A，AULAKH H S，MURALA S.Ri-GAN：an end-to-end network for single images haze removal[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops（CVPRW），Long Beach，USA，2019：2014-2023.
[8] SINGH A，BHAVE A，PRASAD D K.Single images dehazing for a variety of haze scenarios using back projected pyramid network[C]//European Conference on Computer Vision，2020：166-181.
[9] DAS S D，DUTTA S.Fast deep multi-patch hierarchical network for nonhomogeneous images dehazing[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops（CVPRW），Seattle，USA，2020：1994-2001.
[10] YU Y，LIU H，FU M，et al.A two-branch neural network for non-homogeneous dehazing via ensemble learning[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops（CVPRW），Nashville，USA，2021：193-202.
[11] ZACHARIAS J.NTIRE 2021 NonHomogeneous dehazing challenge report[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops（CVPRW），June 19-25，2021：627-646.
[12] 刘春月.基于深度学习的沙尘图像色彩恢复与增强方法研究[D].西安：西安理工大学，2021.
LIU C Y.Color restoration and enhancement of sand dust image based on deep learning[D].Xi’an：Xi’an University of Technology，2021.
[13] GAO G X，LAI H C，LIU Y Q，et al.Sandstorm image enhancement based on YUV space[J].Optik-International Journal for Light and Electron Optics，2021，226：165659.
[14] XU G，WANG X，XU X.Single image enhancement in sandstorm weather via tensor least square[J].Journal of Automatica Sinica，2020，7（6）：1649-1661.
[15] GASPARINI F，SCHETTINI R.Color correction for digital photographs[C]//12th International Conference on Image Analysis and Processing，Mantova，Italy，2003：646-651.
[16] 周荣政，何捷，洪志良.自适应的数码相机自动白平衡算法[J].计算机辅助设计与图形学学报，2005，17（3）：529-533.
ZHOU R Z，HE J，HONG Z L.Adaptive algorithm of auto white balance for digital camera[J].Journal of Compute-Aided Design and Computer Graphics，2005，17（3）：529-533.
[17] 江兴方，金龙，何隆华，等.基于Retinex彩色图像增强及其闭合区域提取的研究[J].光子学报，2007，36（3）：565-567.
JIANG X F，JIN L，HE L H，et al.Research of the color remote sensing image fusing method based on Retinex and edge extraction[J].Acta Photonica Sinica，2007，36（3）：565-567.
[18] LIU H，LI C，WAN Y Q，et al.Dust images enhancement algorithm based on color transfer[C]//Proceedings of the CCF Chinese Conference on Computer Vision，Tianjin，China，2017：168-179.
[19] WU H，LIU J，XIE Y，et al.Knowledge transfer dehazing network for NonHomogeneous dehazing[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops（CVPRW），Seattle，USA，2020：1975-1983.
[20] WANG Z M，WANG J S，YANG K，et al.Semantic segmentation of high-resolution remote sensing images based on a class feature attention mechanism fused with Deeplabv3+[J].Computers & geosciences，2022，158（1）：104969.
[21] SHI W，CABALLERO J，HUSZáR F，et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition（CVPR），Las Vegas，USA，2016：1874-1883.
[22] RUSSAKOVSKY O，DENG J，SU H，et al.ImagesNet large scale visual recognition challenge[J].International Journal of Computer Vision，2015，115（3）：211-252.
[23] ZHANG Y L，LI K P，LI K，et al.Images super-resolution using very deep residual channel attention networks[C]//European Conference on Computer Vision，2018：294-310.
[24] CHEN Jingwen，CHEN Jiawei，CHAO Hongyang，et al.Image blind denoising with generative adversarial network based noise modeling[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），Salt Lake City，USA，2018：3155-3164.
[25] GIRSHICK R.Fast R-CNN[J].arXiv：1504.08083，2015.
[26] ZHU J Y，PARK T，ISOLA P，et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//2017 IEEE International Conference on Computer Vision（ICCV），Venice，Italy，2017：2242-2251.
[27] GUO S，YAN Z，ZHANG K，et al.Toward convolutional blind denoising of real photographs[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），Long Beach，USA，2019：1712-1722.
[28] WANG X，HAN X，HUANG W，et al.Multi-similarity loss with general pair weighting for deep metric learning[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），Long Beach，USA，2019：5017-5025.
[29] ANCUTI C O，ANCUTI C，VASLUIANU F A，et al.NTIRE 2020 challenge on NonHomogeneous dehazing[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops（CVPRW），June 14-19，2020：2029-2044.
[30] MANU C M，SREENI K G.GANID：a novel generative adversarial network for image dehazing[J].The Visual Computer，2022，39：3923-3936.
[31] ANCUTI C O，ANCUTI C，SBERT M，et al.Dense haze：a benchmark for image dehazing with dense-haze and haze-free images[C]//2019 IEEE International Conference on Image Processing（ICIP），Sept 22-25，2019：1014-1018.
[32] ANCUTI C O，ANCUTI C，TIMOFTE R，et al.O-HAZE：a dehazing benchmark with real hazy and haze-free outdoor images[C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops（CVPRW），June 18-22，2018.
[33] EIGEN D，PUHRSCH C，FERGUS R.Depth map prediction from a single image using a multi-scale deep network[C]//International Conference on Neural Information Processing Systems.[S.l.]：MIT Press，Montreal，Canada，2014：2366-2374.
[34] KINGMA D，BA J.Adam：a method for stochastic optimization[J].arXiv：1412.6980，2014.
[35] SHI Z，FENG Y，ZHAO M，et al.Let you see in sand dust weather：a method based on halo-reduced dark channel prior dehazing for sand-dust images enhancement[J].IEEE Access，2019，7：116722-116733.
[36] PARK T H，EOM I K.Sand-dust images enhancement using successive color balance with coincident chromatic histogram[J].IEEE Access，2021，9：19749-19760.
[37] JOBSON D J，RAHMAN Z，WOODELL G A.A multiscale retinexfor bridging the gap between color images and the human observation of scenes[J].IEEE Transactions on Images Processing，2002，6（7）：965-976.
[38] QU Y，CHEN Y，HUANG J，et al.Enhanced Pix2pix dehazing network[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），Long Beach，USA，2019：8152-8160.
[39] MITTAL A.Making a “completely blind” images quality analyzer[J].IEEE Signal Processing Letters，2013，20（3）：209-212.
[40] VENKATANATH N，PRANEETH D，CHANDRASEKHAR B，et al.Blind images quality evaluation using perception based features[C]//2015 Twenty First National Conference on Communications（NCC），Mumbai，India，2015：1-6.
[41] MITTAL A，MOORTHY A K，BOVIK A C.No-reference images quality assessment in the spatial domain[J].IEEE Transactions on Images Processing A Publication of the IEEE Signal Processing Society，2012，21（12）：4695-4708.
[42] GU K，LIN W，ZHAI G，et al.No-reference quality metric of contrast-distorted images based on information maximization[J].IEEE Transactions on Cybernetics，2017，47（12）：4559-4565.
[43] IOFFE S，SZEGEDY C.Batch normalization：accelerating deep network training by reducing internal covariate shift[J].arXiv：1502.03167，2015.