Medical Image Fusion Based on Convolutional Neural Network Model

doi:10.3778/j.issn.1002-8331.2011-0320

Abstract

Abstract: A new medical image fusion algorithm based on convolutional neural network（CNN） and weighted least squares（WLS） is proposed in this paper. The main steps of the algorithm are as follows： the source image is decomposed into a base layer and a series of detail layers using a hybrid multiscale decomposition tool consisting of rolling guided filter（RGF） and Gaussian filter（GF）, which can better preserve the scale information and edge information. Based on convolutional neural network, the fusion rule of base layer is given, which can extract image features better, and the fused image can inherit structure information, energy information and intensity information of the source image well. Combining the “max-absolute” fusion rule with the weighted least squares optimization strategy, the fusion rule of details layer is given to make the fused image contain more visual details and have higher contrast. The proposed algorithm has better advantages than other methods in visual and objective evaluation, and is more effective in image fusion for acute stroke, fatal stroke and meningioma.

Key words: convolutional neural network, weighted least squares, image fusion

摘要： 提出了一种新的基于卷积神经网络（CNN）和加权最小二乘法（WLS）的医学图像融合算法。算法主要步骤如下：利用滚动导向滤波（RGF）和高斯滤波（GF）构成的混合多尺度分解工具将源图像分解为基础层和一系列细节层，从而能够更好地保留尺度信息和边缘信息。基于卷积神经网络给出基础层融合规则，该规则能够更好地提取图像特征，使融合图像能够很好继承源图像结构信息、能量信息和强度信息。利用绝对值取大规则和加权最小二乘法优化策略，对细节层进行融合，使融合图像中包含更多的视觉细节信息和具有更高对比度。实验结果表明所提算法在视觉评价和客观评价方面与其他算法相比具有较好的优势，且在急性中风、致命性中风和脑膜瘤这三类疾病图像融合效果更为突出。

关键词: 卷积神经网络, 加权最小二乘法, 图像融合

LI Yuchen, HUANG Yongdong. Medical Image Fusion Based on Convolutional Neural Network Model[J]. Computer Engineering and Applications, 2022, 58(15): 229-237.

李雨晨, 黄永东. 基于卷积神经网络模型的医学图像融合[J]. 计算机工程与应用, 2022, 58(15): 229-237.

References

[1] DU J，LI W，LU K，et al.An overview of multi-modal medical image fusion[J].Neurocomputing，2016，215：3-20.
[2] JAMES A P，DASARATHY B V.Medical image fusion：a survey of the state of the art[J].Information Fusion，2014，19：4-19.
[3] CUNHA D，ARTHUR L，ZHOU J，et al.The nonsubsampled contourlet transform：theory，design，and applications[J].IEEE Transactions on Image Processing，2006，15（10）：3089-3101.
[4] EASLEY G，LABATE D，LIM W Q.Sparse directional image representations using the discrete shearlet transform[J].Applied and Computational Harmonic Analysis，2008，25（1）：25-46.
[5] LI S，KANG X，HU J.Image fusion with guided filtering[J].IEEE Transactions on Image Processing，2013，22（7）：2864-2875.
[6] TOMASI C，MANDUCHI R.Bilateral filtering for gray and color images[C]//Sixth International Conference on Computer Vision，1998.
[7] DU J，LI W，TAN H.Three-layer medical image fusion with tensor-based features[J].Information Sciences，2020，525.
[8] WANG G F，HUANG Y D.Medical image fusion algorithm based on detail enhancement and parallel feature stimulation pulse coupled neural network[J].Scientia Sinica（Series F：Informationis），2020，50（2）：239-260.
[9] LIU Y，LIU S，WANG Z.A general framework for image fusion based on multi-scale transform and sparse representation[J].Information Fusion，2015，24：147-164.
[10] YAO R，LIU L M，FENG Q S，et al.Medical image fusion based on local energy and PCNN[J].Microcomputer Information，2011，27（10）：155-158.
[11] LIU Y，CHEN X，PENG H，et al.Multi-focus image fusion with a deep convolutional neural network[J].Information Fusion，2017，36：191-207.
[12] ZHANG Q，SHEN X，XU L，et al.Rolling guidance filter[C]//European Conference on Computer Vision，2014.
[13] ZHAO C，HUANG Y D.Infrared and visible image fusion via rolling guidance filtering and hybrid multi-scale decomposition[J].Laser & Optoelectronics Progress，2019，56（14）.
[14] PRABHAKAR K R，SRIKAR V S，BABU R V.DeepFuse：a deep unsupervised approach for exposure fusion with extreme exposure image pairs[C]//IEEE International Conference on Computer Vision，2017.
[15] LI H，WU X J.DenseFuse：a fusion approach to infrared and visible images[J].IEEE Transactions on Image Processing，2019，28（5）：2614-2623.
[16] ZHANG Y，LIU Y，SUN P，et al.IFCNN：a general image fusion framework based on convolutional neural network[J].Information Fusion，2020，54：99-118.
[17] HE K，SUN J，TANG X.Guided image filtering[J].IEEE Transactions on Pattern Analysis & Machine Intelligence，2013，35（6）：1397-1409.
[18] FARBMAN Z，FATTAL R，LISCHINSKI D，et al.Edge-preserving decompositions for multi-scale tone and detail manipulation[J].ACM Transactions on Graphics，2008，27（3）：1-11.
[19] LI S，KANG X，HU J.Image fusion with guided filtering[J].IEEE Transactions on Image Processing，2013，22（7）：2864-2875.
[20] ZHOU Z，WANG B，LI S，et al.Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with gaussian and bilateral filters[J].Information Fusion，2016，30：15-26.
[21] YIN M，LIU X，LIU Y，et al.Medical image fusion with parameter-adaptive pulse coupled neural network in nonsubsampled shearlet transform domain[J].IEEE Transactions on Instrumentation and Measurement，2018，68（1）：49-64.
[22] BURT P J，ADELSON E H.The laplacian pyramid as a compact image code[M]//Readings in computer vision.[S.l.]：Morgan Kaufmann Publishers Inc，1987.
[23] YANG Y，QUE Y，HUANG S，et al.Multimodal sensor medical image fusion based on Type-2 fuzzy logic in NSCT domain[J].IEEE Sensors Journal，2016，16（10）：3735-3745.