Multi-focus Image Fusion Algorithm Based on Unsupervised Deep Learning Model

doi:10.3778/j.issn.1002-8331.2007-0034

Abstract

Abstract:

When deep learning technology is applied to the field of multi-focus image fusion, it mostly trains the network by supervised learning. However, due to the lack of labeled data sets of multi-focus image fusion for supervised training, and the cost of making a dedicated large-scale labeled training set is too high. The existing methods mostly simulate multi-focus image for supervised learning by adding Gaussian blur to the focused image, which makes network training difficult and difficult to achieve the ideal fusion effect. In society to solve the above problems, a multi-focus image fusion method with easy implementation and good fusion effect is proposed. Encoder-Decoder network model with attention mechanism is introduced by training in an unsupervised learning way on easily accessible unlabeled data set to obtain the rich features of the input source images. And then the initial decision map is generated by measuring the activity level of these features through multi-scale morphological focus detection. Eventually, the consistency verification method is used to adjust and optimize the initial decision map, and the final decision map is obtained. The quality of the fusion image is valued on both subjective vision and objective indicators. The experimental results demonstrate that the fusion image has high definition, rich details and low distortion.

Key words: deep learning, multi-focus image fusion, supervised learning, unsupervised learning, morphological focus detection

摘要：

深度学习技术应用到多聚焦图像融合领域时，其大多通过监督学习的方式来训练网络，但由于缺乏专用于多聚焦图像融合的监督训练的标记数据集，且制作专用的大规模标记训练集代价过高，所以现有方法多通过在聚焦图像中随机添加高斯模糊进行监督学习，这导致网络训练难度大，很难实现理想的融合效果。为解决以上问题，提出了一种易实现且融合效果好的多聚焦图像融合方法。通过在易获取的无标记数据集上以无监督学习方式训练引入了注意力机制的encoder-decoder网络模型，获得输入源图像的深层特征。再通过形态聚焦检测对获取的特征进行活动水平测量生成初始决策图。运用一致性验证方法对初始决策图优化，得到最终的决策图。融合图像质量在主观视觉和客观指标两方面上进行评定，经实验结果表明，融合图像清晰度高，保有细节丰富且失真度小。

关键词: 深度学习, 多聚焦图像融合, 监督学习, 无监督学习, 形态聚焦检测

WANG Changcheng, ZHOU Dongming, LIU Yanyu, XIE Shidong. Multi-focus Image Fusion Algorithm Based on Unsupervised Deep Learning Model[J]. Computer Engineering and Applications, 2021, 57(21): 209-215.

王长城，周冬明，刘琰煜，谢诗冬. 无监督深度学习模型的多聚焦图像融合算法[J]. 计算机工程与应用, 2021, 57(21): 209-215.

References

[1] ZHANG Y，LIU Y，SUN P，et al.IFCNN：a general image fusion framework based on convolutional neural network[J].Information Fusion，2020，54：99-118.
[2] LI S，KANG X，FANG L，et al.Pixel-level image fusion：a survey of the state of the art[J].Information Fusion，2017，33：100-112.
[3] LIU Y，CHEN X，PENG H，et al.Multi-focus image fusion with a deep convolutional neural network[J].Information Fusion，2017，36：191-207.
[4] TANG H，XIAO B，LI W，et al.Pixel convolutional neural network for multi-focus image fusion[J].Information Sciences，2017，433/434：125-141.
[5] PRABHAKAR K R，SRIKAR V S，BABU R V.DeepFuse：a deep unsupervised approach for exposure fusion with extreme exposure image pairs[C]//2017 IEEE International Conference on Computer Vision（ICCV），2018：4724-4732.
[6] LI H，WU X J.DenseFuse：a fusion approach to infrared and visible images[J].IEEE Transactions on Image Processing，2019，28：2614-2623.
[7] SHREYAMSHA KUMAR B K.Image fusion based on pixel significance using cross bilateral filter[J].Signal，Image and Video Processing，2013，9（5）：1193-1204.
[8] SHREYAMSHA KUMAR B K.Multifocus and multispectral image fusion based on pixel significance using discrete cosine harmonic wavelet transform[J].Signal，Image and Video Processing，2012，7（6）：1125-1143.
[9] ZHANG Q，GUO B L.Multifocus image fusion using the nonsubsampled contourlet transform[J].Signal Processing，2009，89（7）：1334-1346.
[10] NENCINI F，GARZELLI A，BARONTI S，et al.Remote sensing image fusion using the curvelet transform[J].Information Fusion，2007，8（2）：143-156.
[11] LEWIS J J，O’CALLAGHAN R J，NIKOLOV S G，et al.Pixel-and region-based image fusion with complex wavelets[J].Information Fusion，2007，8（2）：119-130.
[12] ZHANG Y，BAI X，WANG T.Boundary finding based multi-focus image fusion through multi-scale morphological focus?measure[J].Information Fusion，2017，35：81-101.
[13] LI S，KANG X，HU J.Image fusion with guided filtering[J].IEEE Transactions on Image Processing，2013，22（7）：2864-2875.
[14] ZHOU Z，LI S，WANG B.Multi-scale weighted gradient-based fusion for multi-focus images[J].Information Fusion，2014，20：60-72.
[15] LIU Y，LIU S，WANG Z.Multi-focus image fusion with dense SIFT[J].Information Fusion，2015，23：139-155.
[16] HU J，SHEN L，ALBANIE S，et al.Squeeze-and-excitation networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2018，42（8）：2011-2023.
[17] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision & Pattern Recognition，2017.
[18] LIN T Y，MAIRE M，BELONGIE S，et al.Microsoft COCO：common objects in context[C]//Proceedings of the European Conference on Computer Vision，2014.
[19] BAI X，ZHANG Y，ZHOU F，et al.Quadtree-based multi-focus image fusion using a weighted focus-measure[J].Information Fusion，2015，22：105-118.
[20] MA J，ZHOU Z，BO W，et al.Multi-focus image fusion based on multi-scale focus measures and generalized random walk[C]//Proceedings of the 2017 36th Chinese Control Conference（CCC），2017.
[21] AMIN-NAJI M，AGHAGOLZADEH A，EZOJI M.Ensemble of CNN for multi-focus image fusion[J].Information Fusion，2019，51：201-214.