Review of Feature-Level Infrared and Visible Image Fusion

doi:10.3778/j.issn.1002-8331.2403-0157

Abstract

Abstract: Infrared and visible image fusion is an important image processing technique that is widely used because of its practicality. Infrared and visible image fusion (IVIF) is an important branch of multimodal image fusion technology. Based on the research of infrared and visible fusion methods at home and abroad, the basic theory of infrared and visible fusion is described, the current status of IVIF technology is summarized, and the advantages and disadvantages of the traditional method and the deep learning fusion method are analyzed. The detailed summary and analysis are conducted on the IVIF deep learning method, and the existing datasets and fusion performance evaluation indexes are sorted out. Finally, the current challenging issues facing IVIF are discussed in the context of practical applications, and the future development direction of the field is envisioned.

Key words: infrared and visible, image fusion, fusion method

摘要： 红外与可见光图像融合是一门重要的图像处理技术，因其实用性被广泛应用。红外与可见光图像融合（infrared and visible image fusion，IVIF）是多模态图像融合技术中的一个重要分支，在对国内外的红外与可见光融合方法研究基础上，阐述了红外与可见光融合的基本理论，归纳了IVIF技术现状，分析了传统方法和深度学习融合方法的优缺点；重点对IVIF深度学习方法进行了详细的总结和分析，并梳理了现有的数据集和融合性能评价指标。最后，结合实际应用探讨了当前IVIF面临的挑战问题，并对未来该领域的发展方向进行了展望。

关键词: 红外与可见光, 图像融合, 融合方法

ZHANG Honggang, YANG Haitao, ZHENG Fengjie, WANG Jinyu, ZHOU Xixuan, WANG Haoyu, XU Yifan. Review of Feature-Level Infrared and Visible Image Fusion[J]. Computer Engineering and Applications, 2024, 60(18): 17-31.

张宏钢, 杨海涛, 郑逢杰, 王晋宇, 周玺璇, 王浩宇, 徐一帆. 特征级红外与可见光图像融合方法综述[J]. 计算机工程与应用, 2024, 60(18): 17-31.

References

[1] 唐霖峰, 张浩, 徐涵, 等. 基于深度学习的图像融合方法综述[J]. 中国图象图形学报, 2023, 28(1):3-36.
TANG L F, ZHANG H, XU H, et al. Deep learning-based image fusion: a survey[J]. Journal of Image and Graphics, 2023, 28(1):3-36.
[2] LI H, WU X J, DURRANI T. NestFuse: an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9645-9656.
[3] LU Y, WU Y, LIU B, et al. Cross-modality person re-identification with shared-specific feature transfer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 13379-13389.
[4] HA Q, WATANABE K, KARASAWA T, et al. MFNet: towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes[C]//Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017: 5108-5115.
[5] 李扬, 杨海涛, 孔卓, 等. 像素级红外与可见光图像融合方法综述[J]. 计算机工程与应用, 2022, 58(14): 40-50.
LI Y, YANG H T, KONG Z, et al. Review of pixel-level infrared and visible image fusion methods[J]. Computer Engineering and Applications, 2022, 58(14): 40-50.
[6] 刘爽利, 黄雪莉, 刘磊, 等. 光电载荷下的红外和可见光图像融合综述[J]. 计算机工程与应用, 2024, 60(1): 28-39.
LIU S L, HUANG X L, LIU L, et al. Infrared and visible image fusion under photoelectric loads[J]. Computer Engineering and Applications, 2024, 60(1): 28-39.
[7] 安晓东, 李亚丽, 王芳. 汽车驾驶辅助系统红外与可见光融合算法综述[J]. 计算机工程与应用, 2022, 58(19): 64-75.
AN X D, LI Y L, WANG F. Overview of infrared and visible image fusion algorithms for automotive driving assistance system[J]. Computer Engineering and Applications, 2022, 58(19): 64-75.
[8] 刘冬梅. 图像拼接算法研究[D]. 西安: 西安电子科技大学, 2008.
LIU D M. Research on image mosaic algorithm[D]. Xi’an: Xidian University, 2008.
[9] TOET A, HOGERVORST M A. Progress in color night vision[J]. Optical Engineering, 2012, 51(1): 010901.
[10] XU H, MA J, LE Z, et al. FusionDN: a unified densely connected network for image fusion[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2020: 12484-12491.
[11] TANG L, YUAN J, ZHANG H, et al. PIAFusion: a progressive infrared and visible image fusion network based on illumination aware[J]. Information Fusion, 2022, 83: 79-92.
[12] LIU J, FAN X, HUANG Z, et al. Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 5802-5811.
[13] JIA X, ZHU C, LI M, et al. LLVIP: a visible-infrared paired dataset for low-light vision[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 3496-3504.
[14] HWANG S, PARK J, KIM N, et al. Multispectral pedestrian detection: benchmark dataset and baseline[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 1037-1045.
[15] ZHANG M M, CHOI J, DANIILIDIS K, et al. VAIS: a dataset for recognizing maritime imagery in the visible and infrared spectrums[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 10-16.
[16] LI S, KANG X, FANG L, et al. Pixel-level image fusion: a survey of the state of the art[J]. Information Fusion, 2017, 33: 100-112.
[17] ZHANG X, DAI X, ZHANG X, et al. Joint principal component analysis and total variation for infrared and visible image fusion[J]. Infrared Physics & Technology, 2023, 128: 104523.
[18] LU Y, WANG F, LUO X, et al. Novel infrared and visible image fusion method based on independent component analysis[J]. Frontiers of Computer Science, 2014, 8: 243-254.
[19] ZHOU C, ZHAO J, PAN Z, et al. Fusion of visible and infrared images based on IHS transformation and regional variance matching degree[C]//Proceedings of the IOP Conference Series: Earth and Environmental Science, 2019.
[20] YANG B, LI S. Multifocus image fusion and restoration with sparse representation[J]. IEEE Transactions on Instrumentation and Measurement, 2009, 59(4): 884-892.
[21] YANG Y, ZHANG Y, HUANG S, et al. Infrared and visible image fusion using visual saliency sparse representation and detail injection model[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 70: 1-15.
[22] ZHANG S, HUANG F, LIU B, et al. A multi-modal image fusion framework based on guided filter and sparse representation[J]. Optics and Lasers in Engineering, 2021, 137: 106354.
[23] LI Y, LIU G, BAVIRISETTI D P, et al. Infrared-visible image fusion method based on sparse and prior joint saliency detection and LatLRR-FPDE[J]. Digital Signal Processing, 2023, 134: 103910.
[24] BURT P J, ADELSON E H. Merging images through pattern decomposition[C]//Proceedings of the Applications of Digital Image Processing VIII, 1984.
[25] TOET A. Image fusion by a ratio of low-pass pyramid[J]. Pattern Recognition Letters, 1989, 9(4): 245-253.
[26] 胡学龙, 沈洁. 一种基于中值金字塔的图像融合算法[J]. 微电子学与计算机, 2008, 25(9): 165-167.
HU X L, SHEN J. An algorithm on image fusion based on median pyramid[J]. Microelectronics & Computer, 2008, 25(9): 165-165.
[27] LI S, ZOU Y, WANG G, et al. Infrared and visible image fusion method based on a principal component analysis network and image pyramid[J]. Remote Sensing, 2023, 15(3): 685.
[28] HE D, MENG Y, WANG C. Contrast pyramid based image fusion scheme for infrared image and visible image[C]//Proceedings of the 2011 IEEE International Geoscience and Remote Sensing Symposium, 2011: 597-600.
[29] MALLAT S G. A theory for multiresolution signal decom position: the wavelet representation[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 1989, 11(4): 674-693.
[30] ZHANG B. Study on image fusion based on different fusion rules of wavelet transform[C]//Proceedings of the 2010 3rd International Conference on Advanced Computer Theory and Engineering (ICACTE), 2010: 649- 653.
[31] YAN X, QIN H, LI J, et al. Infrared and visible image fusion with spectral graph wavelet transform[J]. JOSA A, 2015, 32(9): 1643-1652.
[32] PU T, NI G. Contrast-based image fusion using the discrete wavelet transform[J]. Optical Engineering, 2000, 39(8): 2075-2082.
[33] ZHANG H, MA X, TIAN Y. An image fusion method based on curvelet transform and guided filter enhancement[J]. Mathematical Problems in Engineering, 2020, 2020:1-8.
[34] BHATNAGAR G, WU Q M J, LIU Z. Directive contrast based multimodal medical image fusion in NSCT domain[J]. IEEE Transactions on Multimedia, 2013, 15(5): 1014-1024.
[35] KONG W. Technique for gray-scale visual light and infrared image fusion based on non-subsampled shearlet transform[J]. Infrared Physics & Technology, 2014, 63: 110-118.
[36] LI Y, ZHAO H, HU Z, et al. IVFuseNet: fusion of infrared and visible light images for depth prediction[J]. Information Fusion, 2020, 58: 1-12.
[37] HOU R, ZHOU D, NIE R, et al. VIF-Net: an unsupervised framework for infrared and visible image fusion[J]. IEEE Transactions on Computational Imaging, 2020, 6: 640-651.
[38] 陈国洋, 吴小俊, 徐天阳. 基于深度学习的无监督红外图像与可见光图像融合算法[J]. 激光与光电子学进展, 2022, 59(4):151-160.
CHEN G Y, WU X J, XU T Y. Unsupervised Infrared image and visible image fusion algorithm based on deep learning[J]. Laser & Optoelectronics Progress, 2022, 59(4): 151-160.
[39] LONG Y, JIA H, ZHONG Y, et al. RXDNFuse: a aggregated residual dense network for infrared and visible image fusion[J]. Information Fusion, 2021, 69: 128-141.
[40] ZHANG Y, LIUU Y, SUN P, et al. IFCNN: a general image fusion framework based on convolutional neural network[J]. Information Fusion, 2020, 54: 99-118.
[41] LIU R, LIU J, JIANG Z, et al. A bilevel integrated model with data-driven layer ensemble for multi-modality image fusion[J]. IEEE Transactions on Image Processing, 2020, 30: 1261-1274.
[42] CHENG C, XU T, WU X J. MUFusion: a general unsupervised image fusion network based on memory unit[J]. Information Fusion, 2023, 92: 80-92.
[43] LUO D, LIU G, BAVIRISETTI D P, et al. Infrared and visible image fusion based on VPDE model and VGG network[J]. Applied Intelligence, 2023, 53(21): 24739-24764
[44] MA J, TANG L, XU M, et al. STDFusionNet: an infrared and visible image fusion network based on salient target detection[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-13.
[45] TANG L, YUAN J, MA J. Image fusion in the loop of high-level vision tasks: a semantic-aware real-time infrared and visible image fusion network[J]. Information Fusion, 2022, 82: 28-42.
[46] GUO C, FAN D, JIANG Z, et al. MDFN: mask deep fusion network for visible and infrared image fusion without reference ground-truth[J]. Expert Systems with Applications, 2023, 211: 118631.
[47] ZHOU Z, FEI E, MIAO L, et al. A perceptual framework for infrared-visible image fusion based on multiscale structure decomposition and biological vision[J]. Information Fusion, 2023, 93: 174-191.
[48] XU M, TANG L, ZHANG H, et al. Infrared and visible image fusion via parallel scene and texture learning[J]. Pattern Recognition, 2022, 132: 108929.
[49] LI H, CEN Y, LIU Y, et al. Different input resolutions and arbitrary output resolution: a meta learning-based deep framework for infrared and visible image fusion[J]. IEEE Transactions on Image Processing, 2021, 30: 4070-4083.
[50] LIU R S, LIU Z, LIU J Y, et al. Searching a hierarchically aggregated fusion architecture for fast multi-modality image fusion [C]//Proceedings of the 29th ACM International Conference on Multimedia, 2021: 1600-1608.
[51] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Advances in Neural Information Processing Systems, 2014:?2672-2680.
[52] MA J, WEI Y, LIANG P, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48: 11-26.
[53] MA J, LIANG P, YU W, et al. Infrared and visible image fusion via detail preserving adversarial learning[J]. Information Fusion, 2020, 54: 85-98.
[54] YUE J, FANG L, XIA S, et al. Dif-Fusion: towards high color fidelity in infrared and visible image fusion with diffusion models[J]. IEEE Transactions on Image Processing, 2023, 32, 5705-5720.
[55] FU Y, WU X J, DURRAIN T. Image fusion based on generative adversarial network consistent with perception[J]. Information Fusion, 2021, 72: 110-125.
[56] YANG X, HUO H T, LI J, et al. DSG-Fusion: infrared and visible image fusion via generative adversarial networks and guided filter[J]. Expert Systems with Applications, 2022, 200: 116905.
[57] RAO Y, WU D, HAN M, et al. AT-GAN: a generative adversarial network with attention and transition for infrared and visible image fusion[J]. Information Fusion, 2023, 92: 336-349.
[58] MA J, XU H, JIANG J, et al. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995.
[59] ZHOU H, HOU J, ZHANG Y, et al. Unified gradient-and intensity-discriminator generative adversarial network for image fusion[J]. Information Fusion, 2022, 88: 184-201.
[60] LI J, HUO H, LI C, et al. AttentionFGAN: infrared and visible image fusion using attention-based generative adversarial networks[J]. IEEE Transactions on Multimedia, 2020, 23: 1383-1396.
[61] ZHANG H, YUAN J, TIAN X, et al. GAN-FM: infrared and visible image fusion using GAN with full-scale skip connection and dual Markovian discriminators[J]. IEEE Transactions on Computational Imaging, 2021, 7: 1134-1147.
[62] MA J Y, ZHANG H, SHAO Z F, et al. GANMcC: a generative adversarial network with multiclassification constraints for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-14.
[63] YANG Y, LIU J X, HUANG S Y, et al. Infrared and visible image fusion via texture conditional generative adversarial network[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(12): 4771-4783.
[64] 周祎楠, 杨晓敏. 基于 GAN 的红外与可见光图像融合算法[J]. 现代计算机, 2021(16): 94-97.
ZHOU W N, YANG X M. Infrared and visible image fusion based on GAN[J]. Modern Computer, 2021(16): 94-97.
[65] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 4700-4708.
[66] LI H, WU X J. DenseFuse: a fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing, 2018, 28(5): 2614-2623.
[67] 王建中, 徐浩楠, 王洪枫, 等. 基于残差密集块和自编码网络的红外与可见光图像融合[J]. 北京理工大学学报, 2021, 41(10): 1077-1083.
WANG J Z, XU H N, WANG H F, et al. Infrared and visible image fusion based on residual dense block and auto-encoder network[J]. Transactions of Beijing Institute of Technology, 2021, 41(10): 1077-1083.
[68] TANG L, XIANG X, ZHANG H, et al. DIVFusion: darkness-free infrared and visible image fusion[J]. Information Fusion, 2023, 91: 477-493.
[69] XU H, GONG M, TIAN X, et al. CUFD: an encoder-decoder network for visible and infrared image fusion based on common and unique feature decomposition[J]. Computer Vision and Image Understanding, 2022, 218: 103407.
[70] REN L, PAN Z B, CAO J Z, et al. Infrared and visible image fusion based on variational auto-encoder and infrared feature compensation[J]. Infrared Physics & Technology, 2021, 117: 103839.
[71] WANG Z, WANG J, WU Y, et al. UNFusion: a unified multi-scale densely connected network for infrared and visible image fusion[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 32(6): 3360-3374.
[72] WANG C, WU J, ZHU Z, et al. MSFNet: MultiStage fusion network for infrared and visible image fusion[J]. Neurocomputing, 2022, 507: 26-39.
[73] JIAN L H, YANG X M, LIU Z, et al. SEDRFuse: a symmetric encoder-decoder with residual block network for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-15.
[74] CHENG C, SUN C, SUN Y, et al. StyleFuse: an unsupervised network based on style loss function for infrared and visible image fusion[J]. Signal Processing: Image Communication, 2022, 106: 116722.
[75] ZHAO Z, XU S, ZHANG C, et al. DIDFuse: deep image decomposition for infrared and visible image fusion[J]. arXiv:2003.09210, 2020.
[76] CHENG C Y, WU X J, XU T Y, et al. UNIFusion: a lightweight unified image fusion network[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-14.
[77] ZHANG G, NIE R, CAO J. SSL-WAEIE: self-supervised learning with weighted auto-encoding and information exchange for infrared and visible image fusion[J]. IEEE/CAA Journal of Automatica Sinica, 2022, 9(9): 1694-1697.
[78] ZHAO F, ZHAO W, YAO L, et al. Self-supervised feature adaption for infrared and visible image fusion[J]. Information Fusion, 2021, 76: 189-203.
[79] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems, 2017:?5998-6008.
[80] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognition at scale[J]. arXiv:2010.11929, 2020.
[81] LIU Z, LIN Y, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 10012-10022.
[82] TU Z, TALEBI H, ZHANG H, et al. Maxvit: multi-axis vision transformer[C]//Proceedings of the European Conference on Computer Vision, 2022: 459-479.
[83] WANG W, XIE E, LI X, et al. Pyramid vision transformer: a versatile backbone for dense prediction without convolutions[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 568-578.
[84] WANG Z, CHEN Y, SHAO W, et al. SwinFuse: a residual swin transformer fusion network for infrared and visible images[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1-12.
[85] CHANG Z, FENG Z, YANG S, et al. AFT: adaptive fusion transformer for visible and infrared images[J]. IEEE Transactions on Image Processing, 2023, 32: 2077-2092.
[86] JIA S, MIN Z, FU X. Multiscale spatial-spectral transformer network for hyperspectral and multispectral image fusion[J]. Information Fusion, 2023, 96: 117-129.
[87] CHEN J, DING J, YU Y, et al. THFuse: an infrared and visible image fusion network using transformer and hybrid feature extractor[J]. Neurocomputing, 2023, 527: 71-82.
[88] LIU Q, PI J, GAO P, et al. STFNet: self-supervised transformer for infrared and visible image fusion[J]. IEEE Transactions on Emerging Topics in Computational Intelligence, 2024, 8(2):?1513-1526.
[89] LI H, WU X J. CrossFuse: a novel cross attention mechanism based infrared and visible image fusion approach[J]. Information Fusion, 2024, 103: 102147.
[90] ZHAO W, XIE S, ZHAO F, et al. MetaFusion: infrared and visible image fusion via meta-feature embedding from object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 13955-13965.
[91] VS V, VALANARASU J M J, OZA P, et al. Image fusion transformer[C]//Proceedings of the 2022 IEEE International Conference on Image Processing (ICIP), 2022: 3566-3570.
[92] LIU X, GAO H, MIAO Q, et al. MFST: multi-modal feature self-adaptive transformer for infrared and visible image fusion[J]. Remote Sensing, 2022, 14(13): 3233.
[93] TANG W, HE F, LIU Y, et al. DATFuse: infrared and visible image fusion via dual attention transformer[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(7):?3159-3172?.
[94] TANG W, HE F, LIU Y. TCCFusion: an infrared and visible image fusion method based on transformer and cross correlation[J]. Pattern Recognition, 2023, 137: 109295.
[95] ZHAO Z, BAI H, ZHANG J, et al. CDDFuse: correlation-driven dual-branch feature decomposition for multi-modality image fusion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 5906-5916.