Image Shadow Removal Algorithm Based on Attention and Multi-Scale Fusion

doi:10.3778/j.issn.1002-8331.2101-0092

Abstract

Abstract: In order to solve the problem of incomplete shadow removal for complex objects or dark areas with similar texture to shadow areas, an image shadow removal algorithm based on attention and multi-scale fusion is proposed. The algorithm is based on the framework of generative adversarial network. Firstly, the user-defined hole residual block is used for feature extraction to obtain accurate shadow feature information and input into the attention guided coding network. Secondly, multi-scale feature fusion is carried out in the encoding stage of attention guidance to obtain different levels of receptive field, so that the coder can capture context information at different scales. Finally, the multiple attention mechanism is used to guide the discriminant network to identify the unshadowed image, thus reducing the loss of key information of the discrimination network and improving the identification effect of the discrimination network. The algorithm is tested on ISTD（dataset with image shadow triplets） and SRD（dataset for shadow removal） public datasets respectively. The results show that the algorithm has good visual effect. Comparing the single shadow removed image with the real unshadowed image in the dataset, SSIM（structural similarity） can reach 0.978, PSNR（peak signal to noise ratio） can reach 32.2 dB, RMSE（root mean squared error） can reach 6.2. Compared with the similar algorithms, it has significant advantages, and has good shadow removal effect for complex objects or dark areas.

Key words: attention mechanism, multi-scale feature fusion, generative adversarial networks, shadow removal

摘要： 针对图像阴影去除算法中复杂地物或与阴影区域纹理相似的暗区域阴影去除不完全的问题，提出了一种注意力与多尺度融合的图像阴影去除算法。该算法基于生成对抗网络框架构建。利用自定义的空洞残差块进行特征提取，获得精确的阴影特征信息并输入到注意力引导的编码网络；在注意力引导的编码阶段进行多尺度的特征融合，获取不同层次的感受野，使编码器能够在不同尺度上捕捉上下文信息；利用多重注意力机制引导判别网络对生成的无阴影图像进行鉴别，进而减少判别网络关键信息损失，提高判别网络的鉴别效果。分别在ISTD（dataset with image shadow triplets）与SRD（dataset for shadow removal）公开数据集上进行验证，实验结果表明：该算法视觉效果表现良好，单幅阴影去除后的图片与数据集中真实无阴影图片进行对比，SSIM（structural similarity）可达到0.978，PSNR（peak signal to noise ratio）可达到32.2?dB，RMSE（root mean squared error）可达到6.2，相比同类算法，具有显著优势，且对复杂地物或暗区域阴影去除效果良好。

关键词: 注意力机制, 多尺度特征融合, 生成对抗网络, 阴影去除

QU Haicheng, TONG Chang, LIU Wanjun. Image Shadow Removal Algorithm Based on Attention and Multi-Scale Fusion[J]. Computer Engineering and Applications, 2022, 58(16): 234-241.

曲海成, 佟畅, 刘万军. 注意力与多尺度融合的图像阴影去除算法[J]. 计算机工程与应用, 2022, 58(16): 234-241.

References

[1] YU X M，LI G，YING Z Q，et al.A new shadow removal method using color-lines[C]//17th International Conference on Computer Analysis of Images and Patterns，2017：307-319.
[2] HUANG X，HUA G，TUMBLIN J，et al.What characterizes a shadow boundary under the sun and sky[C]//2011 IEEE International Conference on Computer Vision，Barcelona，2011：898-905.
[3] FINLAYSON G D，HORDLEY S D，LU C，et al.On the removal of shadows from images[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2006，28（1）：59-68.
[4] GUO R，DAI Q，HOIEM D.Single-image shadow detection and removal using paired regions[C]//2011 IEEE Conference on Computer Vision and Pattern Recognition，2011：2033-2040.
[5] 林雨准，张保明，郭海涛，等.结合多尺度分割和形态学运算的高分辨率遥感影像阴影检测[J].中国图象图形学报，2018，23（8）：1263-1272.
LIN Y Z，ZHANG B M，GUO H T，et al.Shadow detection from high resolution remote sensing imagery based on multi-scale segmentation and morphology operation[J].Journal of Image and Graphics，2018，23（8）：1263-1272.
[6] FAN X Y，WU W J，ZHANG L，et al.Shading-aware shadow detection and removal from a single image[J].The Visual Computer，2020，36（10）：2175-2188.
[7] 焦俊男，石静，田庆久，等.多光谱影像NDVI阴影影响去除模型[J].遥感学报，2020，24（1）：53-66.
JIAO J N，SHI J，TIAN Q J，et al.Research on multispectral-image-based NDVI shadow-effect-eliminating model[J].Journal of Remote Sensing，2020，24（1）：53-66.
[8] ZHANG Y D，CHEN G，VUKOMANOVIC J，et al.Recurrent shadow attention model（RSAM） for shadow removal in high-resolution urban land-cover mapping[J].Remote Sen-
sing of Environment，2020，247：111945.
[9] 吴文，万毅.基于低尺度细节恢复的单幅图像阴影去除方法[J].电子学报，2020，48（7）：1293-1302.
WU W，WAN Y.Shadow removal method for single image based on low scale detail restoration[J].Acta Electronica Sinica，2020，48（7）：1293-1302.
[10] HOSSEINZADEH S，SHAKERI M，ZHANG H.Fast shadow detection from a single image using a patched convolutional neural network[C]//2018 IEEE/RSJ International Conference on Intelligent Robots and Systems，2018：3124-3129.
[11] 张德干，陈晨，董悦，等.一种基于机器学习的运动目标阴影检测新方法[J].光电子·激光，2018，29（12）：1317-1324.
ZHANG D G，CHEN C，DONG Y，et al.A new method of moving object shadow detection based on machine learning[J].Photoelectron Laser，2018，29（12）：1317-1324.
[12] HU X W，JIANG Y T，FU C W，et al.Mask-ShadowGAN：learning to remove shadows from unpaired data[C]//2019 IEEE International Conference on Computer Vision，2019：2472-2481.
[13] FAN H，HAN M，LI J J.Image shadow removal using end-to-end deep convolutional neural networks[J].Applied Sciences，2019，9（5）：1009.
[14] HU X W，FU C W，ZHU L，et al.Direction-aware spatial context features for shadow detection and removal[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2020，42（11）：2795-2808.
[15] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial nets[C]//Advances in Neural Information Processing Systems 27，Montréal，Dec 8-13，2014：2672-2680.
[16] 史彩娟，涂冬景，刘靖祎.Re-GAN：残差生成式对抗网络算法[J].中国图象图形学报，2021，26（3）：594-604.
SHI C J，TU D J，LIU J Y.Re-GAN：residual generative adversarial network algorithm[J].Journal of Image and Graphics，2021，26（3）：594-604.
[17] 张瑞倩，邵振峰，PORTNOV A，等.多尺度空洞卷积的无人机影像目标检测方法[J].武汉大学学报（信息科学版），2020，45（6）：895-903.
ZHANG R Q，SHAO Z F，PORTNOV A，et al.Multi-scale dilated convolutional neural network for object detection in UAV images[J].Geomatics and Information Science of Wuhan University，2020，45（6）：895-903.
[18] XIONG W，XIONG Z，CUI Y，et al.Deep multi-feature fusion network for remote sensing images[J].Remote Sensing Letters，2020，11（6）：563-571.
[19] 任坤，黄泷，范春奇，等.基于多尺度像素特征融合的实时小交通标志检测算法[J].信号处理，2020，36（9）：1457-1463.
REN K，HUANG L，FAN C Q，et al.Real-time small traffic sign detection algorithm based on multi-scale pixel feature fusion[J].Journal of Signal Processing，2020，36（9）：1457-1463.
[20] 卞明运，彭勃，王伟，等.基于空洞卷积的低质量人脸深度伪造图片检测[J].现代电子技术，2021，44（6）：133-138.
BIAN M Y，PENG B，WANG W，et al.Detection of low quality facial deepfake image based on void convolution[J].Modern Electronic Technology，2021，44（6）：133-138.
[21] 贾宇峰，马力.条件约束下的自我注意生成对抗网络[J].西安电子科技大学学报（自然科学版），2019，46（6）：163-170.
JIA Y F，MA L.Self-attention generative adversarial network with the conditional constraint[J].Journal of Xidian University(Natural Science)，2019，46（6）：163-170.
[22] QIAN R，TAN R T，YANG W，et al.Attentive generative adversarial network for raindrop removal from a single image[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition，2018：2482-2491.
[23] 刘烨，黄金筱，马于涛.基于混合神经网络和注意力机制的软件缺陷自动分派方法[J].计算机研究与发展，2020，57（3）：461-473.
LIU Y，HUANG J X，MA Y T.An automatic method using hybrid neural networks and attention mechanism for software bug triaging[J].Journal of Computer Research and Development，2020，57（3）：461-473.
[24] WANG J，LI X，YANG J，et al.Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal[C]//2018 IEEE Conference on Computer Vision and Pattern Recognition，2018：1788-1797.
[25] QU L，TIAN J，HE S，et al.DeshadowNet：a multi-context embedding deep network for shadow removal[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition，2017：2308-2316.
[26] DING B，LONG C J，ZHANG L，et al.ARGAN：attentive recurrent generative adversarial network for shadow detection and removal[C]//2019 IEEE/CVF International Conference on Computer Vision，2019：10212-10221.