Low-Light Image Enhancement Combining Two-Branch Structure and Unpaired GAN

doi:10.3778/j.issn.1002-8331.2205-0524

Abstract

Abstract: In order to solve the problems of edge blurring, inadequate noise suppression and dependence on paired data in most low-light image enhancement methods based on CNN, this paper introduces a two-branch structure into the generator, which allows end-to-end training with unpaired low-light images and normal-light images. The first branch network uses a network similar to the U-Net structure to learn the context feature mapping from low-light image to normal-light image, while the second branch network preserves the details of the original image with full resolution of the low-light image. Finally, the results of the two branch networks are fused by the fusion layer to obtain the final brightened image. At the same time, total variation loss is added to suppress image noise. Qualitative comparisons and quantitative experiments are conducted on six public datasets（MEF, LIME, NPE, VV, ExDark, LOL）. The experimental results show that the proposed method outperforms the other comparison algorithms in the three benchmarks, BRISQUE, NIQE and PIQE, with average values of 17.55, 3.74 and 8.45, respectively. The enhanced image edge details are clear and the image noise is reduced.

Key words: low-light, image enhancement, generative adversarial network, unsupervised learning, two-branch structure

摘要： 为了解决大部分基于CNN的低光图像增强方法边缘模糊、噪声抑制不足以及对配对数据的依赖等问题，使用生成对抗网络，在生成器上引入双分支结构，可在低光图像和正常光图像无配对的情况下，进行端到端训练。第一个分支使用类U-Net结构的网络学习低光图像到正常光图像的上下文特征映射，第二个分支以低光图像的全分辨率来保留原图的细节，最后通过一个融合层融合两个分支的结果，获得最终增亮后的图像，同时加入了total variation loss来抑制图像噪声。在六个公开数据集（MEF、LIME、NPE、VV、ExDark、LOL）上进行了定性比较和定量实验。实验结果表明该方法在BRISQUE，NIQE和PIQE三种基准测试中优于其他对比算法，平均值分别为17.55、3.74和8.45。该算法增强后的图像边缘细节清晰，减弱了图像噪声。

关键词: 低光, 图像增强, 生成对抗网络, 无监督学习, 双分支结构

LI Zhijie, CHEN Ming, FENG Guofu. Low-Light Image Enhancement Combining Two-Branch Structure and Unpaired GAN[J]. Computer Engineering and Applications, 2023, 59(19): 201-210.

李治杰, 陈明, 冯国富. 结合双分支结构和无配对GAN的低光图像增强[J]. 计算机工程与应用, 2023, 59(19): 201-210.

References

[1] WEI C，WANG W，YANG W，et al.Deep retinex decomposition for low-light enhancement[J].arXiv：1808.04560，2018.
[2] LAND E H.The retinex theory of color vision[J].Scientific American，1977，237（6）：108-129.
[3] LYU F，LU F，WU J，et al.MBLLEN：low-light image/video enhancement using CNNs[C]//Proceedings of BMVC 2018，2018：220.
[4] ZHANG F，LI Y，YOU S，et al.Learning temporal consistency for low light video enhancement from single images[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：4967-4976.
[5] GUO C，LI C，GUO J，et al.Zero-reference deep curve estimation for low-light image enhancement[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：1780-1789.
[6] LIU R，MA L，ZHANG J，et al.Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：10561-10570.
[7] JIANG Y，GONG X，LIU D，et al.Enlightengan：deep light enhancement without paired supervision[J].IEEE Transactions on Image Processing，2021，30：2340-2349.
[8] YU Y，LIU H，FU M，et al.A Two-branch neural network for non-homogeneous dehazing via ensemble learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：193-202.
[9] RONNEBERGER O，FISCHER P，BROX T.U-Net：convolutional networks for biomedical image segmentation[C]//Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham：Springer，2015：234-241.
[10] HU J，SHEN L，SUN G.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：7132-7141.
[11] WOO S，PARK J，LEE J Y，et al.CBAM：convolutional block attention module[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：3-19.
[12] ISOLA P，ZHU J Y，ZHOU T，et al.Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：1125-1134.
[13] MAO X，LI Q，XIE H，et al.Least squares generative adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2794-2802.
[14] JOLICOEUR-MARTINEAU A.The relativistic discriminator：a key element missing from standard GAN[J].arXiv：1807.00734，2018.
[15] JOHNSON J，ALAHI A，FEI-FEI L.Perceptual losses for real-time style transfer and super-resolution[C]//Proceedings of the European Conference on Computer Vision.Cham：Springer，2016：694-711.
[16] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409.1556，2014.
[17] ALY H A，DUBOIS E.Image up-sampling using total-variation regularization with a new observation model[J].IEEE Transactions on Image Processing，2005，14（10）：1647-1659.
[18] MA K，ZENG K，WANG Z.Perceptual quality assessment for multi-exposure image fusion[J].IEEE Transactions on Image Processing，2015，24（11）：3345-3356.
[19] GUO X，LI Y，LING H.LIME：Low-light image enhancement via illumination map estimation[J].IEEE Transactions on Image Processing，2016，26（2）：982-993.
[20] WANG S，ZHENG J，HU H M，et al.Naturalness preserved enhancement algorithm for non-uniform illumination images[J].IEEE Transactions on Image Processing，2013，22（9）：3538-3548.
[21] LOH Y P，CHAN C S.Getting to know low-light images with the exclusively dark dataset[J].Computer Vision and Image Understanding，2019，178：30-42.
[22] MITTAL A，SOUNDARARAJAN R，BOVIK A C.Making a “completely blind” image quality analyzer[J].IEEE Signal Processing Letters，2012，20（3）：209-212.
[23] MITTAL A，MOORTHY A K，BOVIK A C.No-reference image quality assessment in the spatial domain[J].IEEE Transactions on image Processing，2012，21（12）：4695-4708.
[24] VENKATANATH N，PRANEETH D，CHANDRASEKHAR B，et al.Blind image quality evaluation using perception based features[C]//Proceedings of the 21st National Conference on Communications（NCC），2015.