Review of Pixel-Level Infrared and Visible Image Fusion Methods

doi:10.3778/j.issn.1002-8331.2203-0318

Abstract

Abstract: Multi-source image fusion is widely used as a technique for enhancement. The fusion technique improves the efficiency of subsequent advanced vision tasks or manual recognition by extracting the distinctive features of each of visible and infrared and retaining them in the fused image. On the basis of absorbing the research of many scholars at home and abroad, the definition and classification of image fusion are described through research generalization, the algorithms of multi-source fusion methods are systematically summarized, and the direction of further development of image fusion algorithms is analyzed.

Key words: infrared and visible light, image fusion, fusion method

摘要： 红外与可见光图像融合作为一种针对增强技术被广泛应用，融合技术通过提取可见光和红外各自的显著特征，并将其保留在融合图像中，提高了后续高级视觉任务的效率或人工识别。在吸收国内外众多学者的研究的基础上，通过研究归纳，阐述了图像融合的定义与分类，系统性地总结了红外与可见光融合算法，分析了图像融合算法进一步发展的方向。

关键词: 红外与可见光, 图像融合, 融合方法

LI Yang, YANG Haitao, KONG Zhuo, ZHANG Changgong, WANG Jinyu. Review of Pixel-Level Infrared and Visible Image Fusion Methods[J]. Computer Engineering and Applications, 2022, 58(14): 40-50.

李扬, 杨海涛, 孔卓, 张长弓, 王晋宇. 像素级红外与可见光图像融合方法综述[J]. 计算机工程与应用, 2022, 58(14): 40-50.

References

[1] 郭雷，李晖晖，鲍永生.图像融合[M].北京：电子工业出版社，2008.
GUO L，LI H H，BAO Y S.Image fusion[M].Beijing：Publishing House of Electronics Industry，2008.
[2] 周涛，刘珊，董雅丽，等.多尺度变换像素级医学图像融合：研究进展、应用和挑战[J].中国图象图形学报，2021，26（9）：2094-2110.
ZHOU T，LIU S，DONG Y L，et al.Research on pixel-level image fusion based on multi-scale transformation：progress application and challenges[J].Journal of Image and Graphics，201，26（9）：2094-2110.
[3] 沈英，黄春红，黄峰，等.红外与可见光图像融合技术的研究进展[J].红外与激光工程，2021，50（9）：152-169.
SHEN Y，HUANG C H，HUANG F，et al.Research progress of infrared and visible image fusion technology[J].Infrared and Laser Engineering，2021，50（9）：152-169.
[4] 王耀南，李树涛.多传感器信息融合及其应用综述[J].控制与决策，2001（5）：518-522.
WANG Y N，LI S T.Multisensor information fusion and its application：a survey[J].Control and Decision，2001（5）：518-522.
[5] 王海晖，彭嘉雄，吴巍，等.多源遥感图像融合效果评价方法研究[J].计算机工程与应用，2003，39（25）：33-37.
WANG H H，PENG J X，WU W，et al.A study of evaluation methods on performance of the multi-source remote sensing image fusion[J].Computer Engineering and Applications，2003，39（25）：33-37.
[6] ZHANG X，YE P，XIAO G.VIFB：a visible and infrared image fusion benchmark[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops（CVPRW），2020：468-478.
[7] 刘冬梅.图像拼接算法研究[D].西安：西安电子科技大学，2008.
LIU D M.Research on image mosaic algorithm[D].Xi’an：Xidian University，2008.
[8] TOET A，HOGERVORST M A.Progress in color night vision[J].Optical Engineering，2012，51（1）：010901.
[9] ZHANG M M，CHOI J，DANIILIDIS K，et al.VAIS：a dataset for recognizing maritime imagery in the visible and infrared spectrums[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops（CVPRW），2015.
[10] HWANG S，PARK J，KIM N，et al.Multispectral pedestrian detection：Benchmark dataset and baseline[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition（CVPR），2013.
[11] BURT P J，ADELSON E H.Merging images through pattern decomposition[C]//Applications of Digital Image Processing VIII，1984.
[12] TOET A.Image fusion by a ratio of low-pass pyramid[J].Pattern Recognition Letters，1989，9（4）：245-253.
[13] BURT P J.A gradient pyramid basis for pattern-selective image fusion[C]//Processings of the Society for Information Display Conference，1992.
[14] 陈锦，曾东.基于Spline金字塔的图像融合方法[J].实验科学与技术，2007（1）：129-131.
CHEN J，ZENG D.Image fusion algorithm based on spline pyramid[J].Experiment Science & Technology，2007（1）：129-131.
[15] 胡学龙，沈洁.一种基于中值金字塔的图像融合算法[J].微电子学与计算机，2008（9）：165-167.
HU X L，SHEN J.An algorithm on image fusion based on median pyramid[J].Microelectronics & Computer，2008（9）：165-167.
[16] 崔颢.基于方向可控金字塔的图像融合算法[J].航空计算技术，2011，41（4）：24-27.
CUI H.An image fusion algorithm based on steerable pyramid[J].Aeronautical Computing Technique，2011，41（4）：24-27.
[17] 刘斌，董迪，陈俊霖.基于方向性对比度金字塔的图像融合方法[J].量子电子学报，2017，34（4）：405-413.
LIU B，DONG D，CHEN J L.Image fusion method based on directional contrast pyramid[J].Chinese Journal of Quantum Electronics，2017，34（4）：405-413.
[18] KOU F，LI Z，WEN C，et al.Edge-preserving smoothing pyramid based multi-scale exposure fusion[J].Journal of Visual Communication and Image Representation，2018，53：235-244.
[19] 刘斌，辛迦楠，谌文江，等.不可分拉普拉斯金字塔构造及其在多光谱图像融合中的应用[J].计算机应用，2019，39（2）：564-570.
LIU B，XIN J N，CHEN W J，et al.Construction of non-separable Laplacian pyramid and its application in multi-spectral image fusion[J].Journal of Computer Applications，2019，39（2）：564-570.
[20] MALLAT S G.A theory for multiresolution signal decomposition：the wavelet representation[J].IEEE Transactions on Pattern Analysis & Machine Intelligence，1989，11（4）：674-693.
[21] LI H，MANJUNATH B S，MITRA S K.Multisensor image fusion using the wavelet transform[J].Graphical Models & Image Processing，1995，57（3）：235-245.
[22] 王亚杰，李殿起，徐心和.基于双树复小波变换彩色多聚焦图像融合方法[J].计算机工程与应用，2007，43（28）：12-14.
WANG Y J，LI D Q，XU X H.Color multi-focus image fusion method based on double tree complex wavelet transform[J].Computer Engineering and Applications，2007，43（28）：12-14.
[23] UYTTERHOEVEN G，BULTHEEL A.The red-black wavelet transform[C]//Proceedings of IEEE Benelux Signal Processing Symposium，1997.
[24] KINGSBURY N.The dual-tree complex wavelet transform：a new technique for shift invariance and directional filters[C]//IEEE Digital Signal Processing Workshop，1998.
[25] IOANNIDOU S，KARATHANASSI V.Investigation of the dual-tree complex and shift-invariant discrete wavelet transforms on quickbird image fusion[J].IEEE Geoscience and Remote Sensing Letters，2007，4：166-170.
[26] LEE C S，LEE C K，YOO K Y.New lifting based structure for undecimated wavelet transform[J].Electronics Letters，2000，36（22）：1894-1895.
[27] PIELLA G.A general framework for multiresolution image fusion：from pixels to regions[J].Information Fusion，2003，4（4）：259-280.
[28] BAYRO-CORROCHANO E.The theory and use of the quaternion wavelet transform[J].Journal of Mathematical Imaging and Vision，2006，24（1）：19-35.
[29] CHAI P F，LUO X Q，ZHANG Z C.Image fusion using quaternion wavelet transform and multiple features[J].IEEE Access，2017，5：6724-6734.
[30] 王卫星，曾基兵.冗余提升不可分离小波的图像融合方法[J].电子科技大学学报，2009，38（1）：13-16.
WANG W X，ZENG J B.Image fusion method for non-separable wavelet with redundant lifting[J].Journal of University of Electronic Science and Technology of China，2009，38（1）：13-16.
[31] GILLES J.Empirical wavelet transform[J].IEEE Transactions on Signal Processing，2013，61（16）：3999-4010.
[32] 李雄飞，宋璐，张小利.基于协同经验小波变换的遥感图像融合[J].吉林大学学报（工学版），2019，49（4）：1307-1319.
LI X F，SONG L，ZHANG X L.Remote sensing image fusion based on collaborative empirical wavelet transform[J].Journal of Jilin University（Engineering and Technology Edition），2019，49（4）：1307-1319.
[33] 宫睿，王小春.基于可协调经验小波变换的多聚焦图像融合[J].计算机工程与应用，2020，56（2）：201-210.
GONG R，WANG X C.Multi-focus image fusion based on harmonized empirical wavelet transform[J].Computer Engineering and Applications，2020，56（2）：201-210.
[34] CANDèS E J，DONOHO D L.Ridgelets：a key to higher-dimensional intermittency?[J].Philosophical Transactions of the Royal Society of London，Series A：Mathematical，Physical and Engineering Sciences，1999，357（1760）：2495-2509.
[35] CANDES E J.Ridgelets：theory and applications[R].Stanford University.Department of Statistics，1998.
[36] CHOI M，KIM R Y，NAM M R，et al.Fusion of multispectral and panchromatic satellite images using the curvelet transform[J].IEEE Geoscience & Remote Sensing Letters，2005，2（2）：136-140.
[37] 张强，郭宝龙.一种基于Curvelet变换多传感器图像融合算法[J].光电子·激光，2006（9）：1123-1127.
ZHANG Q，GUO B L.Fusion of multisensor images based on the Curvelet transform[J].Journal of Optoelectronics·Laser，2006（9）：1123-1127.
[38] PENNEC E L，MALLAT S.Sparse geometric image representations with bandelets.[J].IEEE Transactions on Image Processing，2005，14（4）：423-438.
[39] 杨扬，戴明，周箩鱼，等.基于非下采样Bandelet变换的多聚焦图像融合[J].吉林大学学报（工学版），2014，44（2）：525-530.
YANG Y，DAI M，ZHOU L Y，et al.Multi-focus image fusion based on non-subsampled Bandelet transform[J].Journal of Jilin University（Engineering and Technology Edition），2014，44（2）：525-530.
[40] DA CUNHA A L，ZHOU J，DO M N.The nonsubsampled contourlet transform：theory，design，and applications[J].IEEE Transactions on Image Processing，2006，15（10）：3089-3101.
[41] GOYAL B，DOGRA A，KHOOND R，et al.An efficient medical assistive diagnostic algorithm for visualisation of structural and tissue details in CT and MRI fusion[J].Cognitive Computation，2021，13（6）：1471-1483.
[42] 裴高乐，史涛，李世星.NSCT与AR-PCNN相结合的医学图像融合研究[J/OL].激光杂志：1-7[2022-03-30]..http：//kns.cnki.net/kcms/detail/50.1085.TN.20220301.1955.010.html.
PEI G L，SHI T，LI S X.Research on medical image fusion based on NSCT and AR-PCNN[J].Laser Journal：1-7[2022-03-30].http：//kns.cnki.net/kcms/detail/50.1085.TN.20220301.1955.010.html.
[43] GUO K，LABATE D，et al.Optimally sparse multidimensional representation using shearlets[J].SIAM Journal on Mathematical Analysis，2007，39（1）：298-318.
[44] 王鸿闯.基于NSCT域的自适应阈值图像去噪算法研究[D].兰州：兰州交通大学，2018.
WANG H C.Research on adaptive threshold image denoising algorithm based on NSCT domain[D].Lanzhou：Lanzhou Jiaotong University，2018.
[45] EASLEY G，LABATE D，LIM W Q.Sparse directional image representations using the discrete shearlet transform[J].Applied and Computational Harmonic Analysis，2008，25（1）：25-46.
[46] WEI R，ZHU D，ZHAN W，et al.Infrared and visible image fusion based on RPCA and NSST[C]//2019 IEEE International Conference on Power，Intelligent Computing and Systems（ICPICS），2019：236-240.
[47] KROMMWEH J.Tetrolet transform：a new adaptive Haar wavelet algorithm for sparse image representation[J].Journal of Visual Communication and Image Representation，2010，21（4）：364-374.
[48] 苑玉彬，彭静，沈瑜，等.基于Tetrolet变换的近红外与彩色可见光图像融合算法研究[J].红外技术，2020，42（3）：223-230.
YUAN Y B，PENG J，SHEN Y，et al.Research on near-infrared and color visible image fusion algorithm based on Tetrolet transform[J].Infrared Technology，2020，42（3）：223-230.
[49] CHAVAN S S，MAHAJAN A，TALBAR S N，et al.Nonsubsampled rotated complex wavelet transform（NSRCxWT） for medical image fusion related to clinical aspects in neurocysticercosis[J].Computers in Biology & Medicine，2016，81：64.
[50] DOGAN H，BAYKAL E，EKINCI M，et al.A novel extended depth of field process based on nonsubsampled shearlet transform by estimating optimal range in microscopic systems[J].Optics Communications，2018，429：88-99.
[51] CHENG B，JIN L，LI G.Adaptive fusion framework of infrared and visual image using saliency detection and improved dual-channel PCNN in the LNSST domain[J].Infrared Physics & Technology，2018，92：30-43.
[52] 沈瑜，陈小朋，杨倩.多方向Laplacian能量和与tetrolet变换的图像融合[J].中国图象图形学报，2020，25（4）：721-731.
SHEN Y，CHEN X P，YANG Q.Multi-direction Laplacian energy and image fusion with tetrolet transform[J].Journal of Image and Graphics，2020，25（4）：721-731.
[53] CHENG B，JIN L，LI G.Adaptive fusion framework of infrared and visual image using saliency detection and improved dual-channel PCNN in the LNSST domain[J].Infrared Physics & Technology，2018，92：30-43.
[54] MENG F，SONG M，GUO B，et al.Image fusion based on object region detection and non-subsampled contourlet transform[J].Computers & Electrical Engineering，2017，62：375-383.
[55] AISHWARYA N，THANGAMMAL C B.Visible and infrared image fusion using DTCWT and adaptive combined clustered dictionary-ScienceDirect[J].Infrared Physics & Technology，2018，93：300-309.
[56] CHENG B，JIN L，LI G.Adaptive fusion framework of infrared and visual image using saliency detection and improved dual-channel PCNN in the LNSST domain[J].Infrared Physics & Technology，2018，92：30-43.
[57] CAI H，ZHUO L，CHEN X.Infrared and visible image fusion based on BEMSD and improved fuzzy set[J].Infrared Physics & Technology，2019，98：201-211.
[58] LIU X，MEI W，DU H.Multi-modality medical image fusion based on image decomposition framework and nonsubsampled shearlet transform[J].Biomedical Signal Processing and Control，2018，40：343-350.
[59] ULLAH H，ULLAH B，WU L，et al.Multi-modality medical images fusion based on local-features fuzzy sets and novel sum-modified-Laplacian in non-subsampled shearlet transform domain[J].Biomedical Signal Processing and Control，2020，57：101724.
[60] DING W，BI D，HE L，et al.Infrared and visible image fusion method based on sparse features[J].Infrared Physics & Technology，2018，92：372-380.
[61] MENG L，GUO X，LI H.MRI/CT fusion based on latent low rank representation and gradient transfer[J].Biomedical Signal Processing and Control，2019，53：101536.
[62] VISHWAKARMA A，BHUYAN M K，IWAHORI Y.An optimized non-subsampled shearlet transform-based image fusion using Hessian features and unsharp masking[J].Journal of Visual Communication and Image Representation，2018，57：48-60.
[63] ANANDHI D，VALLI S.An algorithm for multi-sensor image fusion using maximum a posteriori and nonsubsampled contourlet transform[J].Computers & Electrical Engineering，2017：139-152.
[64] 刘斌，谌文江，辛迦楠.基于四通道不可分加性小波的多聚焦图像融合[J].计算机科学，2019，46（7）：268-273.
LIU B，CHEN W J，XIN J N.Multifocus image fusion based on four channel non-separable additive wavelet[J].Computer Science，2019，46（7）：268-273.
[65] CHENG B，JIN L，LI G.General fusion method for infrared and visual images via latent low-rank representation and local non-subsampled shearlet transform[J].Infrared Physics & Technology，2018，92：68-77.
[66] LI H，WU X J.DenseFuse：a fusion approach to infrared and visible images[J].IEEE Transactions on Image Processing，2018，28（5）：2614-2623.
[67] HOU R.VIF-Net：an unsupervised framework for infrared and visible image fusion[J].IEEE Transactions on Computational Imaging，2020，6：640-651.
[68] MA J，WEI Y，LIANG P，et al.FusionGAN：a generative adversarial network for infrared and visible image fusion[J].Information Fusion，2019，48：11-26.
[69] MA J，XU H，JIANG J，et al.DDcGAN：a dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J].IEEE Transactions on Image Processing，2020，29：4980-4995.
[70] MA J，YU W，CHEN C，et al.Pan-GAN：an unsupervised pan-sharpening method for remote sensing image fusion[J].Information Fusion，2020，62：110-120.
[71] MA J，ZHANG H，SHAO Z，et al.GANMcC：a generative adversarial network with multiclassification constraints for infrared and visible image fusion[J].IEEE Transactions on Instrumentation and Measurement，2020，70：5005014.
[72] ZHANG H，MA J.SDNet：a versatile squeeze-and-decomposition network for real-time image fusion[J].International Journal of Computer Vision，2021：1-25.
[73] MA J，TANG L，XU M，et al.STDFusionNet：an infrared and visible image fusion network based on salient target detection[J].IEEE Transactions on Instrumentation and Measurement，2021，70：1-13.
[74] H LI，WU X J，DURRANI T.NestFuse：an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models[J].IEEE Transactions on Instrumentation and Measurement，2020，69（12）：9645-9656.
[75] TANG L，YUAN J，MA J.Image fusion in the loop of high-level vision tasks：a semantic-aware real-time infrared and visible image fusion network[J].Information Fusion，2022，82：28-42.