Computer Engineering and Applications ›› 2024, Vol. 60 ›› Issue (18): 17-31.DOI: 10.3778/j.issn.1002-8331.2403-0157
• Research Hotspots and Reviews • Previous Articles Next Articles
ZHANG Honggang, YANG Haitao, ZHENG Fengjie, WANG Jinyu, ZHOU Xixuan, WANG Haoyu, XU Yifan
Online:
2024-09-15
Published:
2024-09-13
张宏钢,杨海涛,郑逢杰,王晋宇,周玺璇,王浩宇,徐一帆
ZHANG Honggang, YANG Haitao, ZHENG Fengjie, WANG Jinyu, ZHOU Xixuan, WANG Haoyu, XU Yifan. Review of Feature-Level Infrared and Visible Image Fusion[J]. Computer Engineering and Applications, 2024, 60(18): 17-31.
张宏钢, 杨海涛, 郑逢杰, 王晋宇, 周玺璇, 王浩宇, 徐一帆. 特征级红外与可见光图像融合方法综述[J]. 计算机工程与应用, 2024, 60(18): 17-31.
Add to citation manager EndNote|Ris|BibTeX
URL: http://cea.ceaj.org/EN/10.3778/j.issn.1002-8331.2403-0157
[1] 唐霖峰, 张浩, 徐涵, 等. 基于深度学习的图像融合方法综述[J]. 中国图象图形学报, 2023, 28(1):3-36. TANG L F, ZHANG H, XU H, et al. Deep learning-based image fusion: a survey[J]. Journal of Image and Graphics, 2023, 28(1):3-36. [2] LI H, WU X J, DURRANI T. NestFuse: an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9645-9656. [3] LU Y, WU Y, LIU B, et al. Cross-modality person re-identification with shared-specific feature transfer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 13379-13389. [4] HA Q, WATANABE K, KARASAWA T, et al. MFNet: towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes[C]//Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017: 5108-5115. [5] 李扬, 杨海涛, 孔卓, 等. 像素级红外与可见光图像融合方法综述[J]. 计算机工程与应用, 2022, 58(14): 40-50. LI Y, YANG H T, KONG Z, et al. Review of pixel-level infrared and visible image fusion methods[J]. Computer Engineering and Applications, 2022, 58(14): 40-50. [6] 刘爽利, 黄雪莉, 刘磊, 等. 光电载荷下的红外和可见光图像融合综述[J]. 计算机工程与应用, 2024, 60(1): 28-39. LIU S L, HUANG X L, LIU L, et al. Infrared and visible image fusion under photoelectric loads[J]. Computer Engineering and Applications, 2024, 60(1): 28-39. [7] 安晓东, 李亚丽, 王芳. 汽车驾驶辅助系统红外与可见光融合算法综述[J]. 计算机工程与应用, 2022, 58(19): 64-75. AN X D, LI Y L, WANG F. Overview of infrared and visible image fusion algorithms for automotive driving assistance system[J]. Computer Engineering and Applications, 2022, 58(19): 64-75. [8] 刘冬梅. 图像拼接算法研究[D]. 西安: 西安电子科技大学, 2008. LIU D M. Research on image mosaic algorithm[D]. Xi’an: Xidian University, 2008. [9] TOET A, HOGERVORST M A. Progress in color night vision[J]. Optical Engineering, 2012, 51(1): 010901. [10] XU H, MA J, LE Z, et al. FusionDN: a unified densely connected network for image fusion[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2020: 12484-12491. [11] TANG L, YUAN J, ZHANG H, et al. PIAFusion: a progressive infrared and visible image fusion network based on illumination aware[J]. Information Fusion, 2022, 83: 79-92. [12] LIU J, FAN X, HUANG Z, et al. Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 5802-5811. [13] JIA X, ZHU C, LI M, et al. LLVIP: a visible-infrared paired dataset for low-light vision[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 3496-3504. [14] HWANG S, PARK J, KIM N, et al. Multispectral pedestrian detection: benchmark dataset and baseline[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 1037-1045. [15] ZHANG M M, CHOI J, DANIILIDIS K, et al. VAIS: a dataset for recognizing maritime imagery in the visible and infrared spectrums[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 10-16. [16] LI S, KANG X, FANG L, et al. Pixel-level image fusion: a survey of the state of the art[J]. Information Fusion, 2017, 33: 100-112. [17] ZHANG X, DAI X, ZHANG X, et al. Joint principal component analysis and total variation for infrared and visible image fusion[J]. Infrared Physics & Technology, 2023, 128: 104523. [18] LU Y, WANG F, LUO X, et al. Novel infrared and visible image fusion method based on independent component analysis[J]. Frontiers of Computer Science, 2014, 8: 243-254. [19] ZHOU C, ZHAO J, PAN Z, et al. Fusion of visible and infrared images based on IHS transformation and regional variance matching degree[C]//Proceedings of the IOP Conference Series: Earth and Environmental Science, 2019. [20] YANG B, LI S. Multifocus image fusion and restoration with sparse representation[J]. IEEE Transactions on Instrumentation and Measurement, 2009, 59(4): 884-892. [21] YANG Y, ZHANG Y, HUANG S, et al. Infrared and visible image fusion using visual saliency sparse representation and detail injection model[J]. IEEE Transactions on Instrumentation and Measurement, 2020, 70: 1-15. [22] ZHANG S, HUANG F, LIU B, et al. A multi-modal image fusion framework based on guided filter and sparse representation[J]. Optics and Lasers in Engineering, 2021, 137: 106354. [23] LI Y, LIU G, BAVIRISETTI D P, et al. Infrared-visible image fusion method based on sparse and prior joint saliency detection and LatLRR-FPDE[J]. Digital Signal Processing, 2023, 134: 103910. [24] BURT P J, ADELSON E H. Merging images through pattern decomposition[C]//Proceedings of the Applications of Digital Image Processing VIII, 1984. [25] TOET A. Image fusion by a ratio of low-pass pyramid[J]. Pattern Recognition Letters, 1989, 9(4): 245-253. [26] 胡学龙, 沈洁. 一种基于中值金字塔的图像融合算法[J]. 微电子学与计算机, 2008, 25(9): 165-167. HU X L, SHEN J. An algorithm on image fusion based on median pyramid[J]. Microelectronics & Computer, 2008, 25(9): 165-165. [27] LI S, ZOU Y, WANG G, et al. Infrared and visible image fusion method based on a principal component analysis network and image pyramid[J]. Remote Sensing, 2023, 15(3): 685. [28] HE D, MENG Y, WANG C. Contrast pyramid based image fusion scheme for infrared image and visible image[C]//Proceedings of the 2011 IEEE International Geoscience and Remote Sensing Symposium, 2011: 597-600. [29] MALLAT S G. A theory for multiresolution signal decom position: the wavelet representation[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 1989, 11(4): 674-693. [30] ZHANG B. Study on image fusion based on different fusion rules of wavelet transform[C]//Proceedings of the 2010 3rd International Conference on Advanced Computer Theory and Engineering (ICACTE), 2010: 649- 653. [31] YAN X, QIN H, LI J, et al. Infrared and visible image fusion with spectral graph wavelet transform[J]. JOSA A, 2015, 32(9): 1643-1652. [32] PU T, NI G. Contrast-based image fusion using the discrete wavelet transform[J]. Optical Engineering, 2000, 39(8): 2075-2082. [33] ZHANG H, MA X, TIAN Y. An image fusion method based on curvelet transform and guided filter enhancement[J]. Mathematical Problems in Engineering, 2020, 2020:1-8. [34] BHATNAGAR G, WU Q M J, LIU Z. Directive contrast based multimodal medical image fusion in NSCT domain[J]. IEEE Transactions on Multimedia, 2013, 15(5): 1014-1024. [35] KONG W. Technique for gray-scale visual light and infrared image fusion based on non-subsampled shearlet transform[J]. Infrared Physics & Technology, 2014, 63: 110-118. [36] LI Y, ZHAO H, HU Z, et al. IVFuseNet: fusion of infrared and visible light images for depth prediction[J]. Information Fusion, 2020, 58: 1-12. [37] HOU R, ZHOU D, NIE R, et al. VIF-Net: an unsupervised framework for infrared and visible image fusion[J]. IEEE Transactions on Computational Imaging, 2020, 6: 640-651. [38] 陈国洋, 吴小俊, 徐天阳. 基于深度学习的无监督红外图像与可见光图像融合算法[J]. 激光与光电子学进展, 2022, 59(4):151-160. CHEN G Y, WU X J, XU T Y. Unsupervised Infrared image and visible image fusion algorithm based on deep learning[J]. Laser & Optoelectronics Progress, 2022, 59(4): 151-160. [39] LONG Y, JIA H, ZHONG Y, et al. RXDNFuse: a aggregated residual dense network for infrared and visible image fusion[J]. Information Fusion, 2021, 69: 128-141. [40] ZHANG Y, LIUU Y, SUN P, et al. IFCNN: a general image fusion framework based on convolutional neural network[J]. Information Fusion, 2020, 54: 99-118. [41] LIU R, LIU J, JIANG Z, et al. A bilevel integrated model with data-driven layer ensemble for multi-modality image fusion[J]. IEEE Transactions on Image Processing, 2020, 30: 1261-1274. [42] CHENG C, XU T, WU X J. MUFusion: a general unsupervised image fusion network based on memory unit[J]. Information Fusion, 2023, 92: 80-92. [43] LUO D, LIU G, BAVIRISETTI D P, et al. Infrared and visible image fusion based on VPDE model and VGG network[J]. Applied Intelligence, 2023, 53(21): 24739-24764 [44] MA J, TANG L, XU M, et al. STDFusionNet: an infrared and visible image fusion network based on salient target detection[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-13. [45] TANG L, YUAN J, MA J. Image fusion in the loop of high-level vision tasks: a semantic-aware real-time infrared and visible image fusion network[J]. Information Fusion, 2022, 82: 28-42. [46] GUO C, FAN D, JIANG Z, et al. MDFN: mask deep fusion network for visible and infrared image fusion without reference ground-truth[J]. Expert Systems with Applications, 2023, 211: 118631. [47] ZHOU Z, FEI E, MIAO L, et al. A perceptual framework for infrared-visible image fusion based on multiscale structure decomposition and biological vision[J]. Information Fusion, 2023, 93: 174-191. [48] XU M, TANG L, ZHANG H, et al. Infrared and visible image fusion via parallel scene and texture learning[J]. Pattern Recognition, 2022, 132: 108929. [49] LI H, CEN Y, LIU Y, et al. Different input resolutions and arbitrary output resolution: a meta learning-based deep framework for infrared and visible image fusion[J]. IEEE Transactions on Image Processing, 2021, 30: 4070-4083. [50] LIU R S, LIU Z, LIU J Y, et al. Searching a hierarchically aggregated fusion architecture for fast multi-modality image fusion [C]//Proceedings of the 29th ACM International Conference on Multimedia, 2021: 1600-1608. [51] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Advances in Neural Information Processing Systems, 2014:?2672-2680. [52] MA J, WEI Y, LIANG P, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48: 11-26. [53] MA J, LIANG P, YU W, et al. Infrared and visible image fusion via detail preserving adversarial learning[J]. Information Fusion, 2020, 54: 85-98. [54] YUE J, FANG L, XIA S, et al. Dif-Fusion: towards high color fidelity in infrared and visible image fusion with diffusion models[J]. IEEE Transactions on Image Processing, 2023, 32, 5705-5720. [55] FU Y, WU X J, DURRAIN T. Image fusion based on generative adversarial network consistent with perception[J]. Information Fusion, 2021, 72: 110-125. [56] YANG X, HUO H T, LI J, et al. DSG-Fusion: infrared and visible image fusion via generative adversarial networks and guided filter[J]. Expert Systems with Applications, 2022, 200: 116905. [57] RAO Y, WU D, HAN M, et al. AT-GAN: a generative adversarial network with attention and transition for infrared and visible image fusion[J]. Information Fusion, 2023, 92: 336-349. [58] MA J, XU H, JIANG J, et al. DDcGAN: a dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995. [59] ZHOU H, HOU J, ZHANG Y, et al. Unified gradient-and intensity-discriminator generative adversarial network for image fusion[J]. Information Fusion, 2022, 88: 184-201. [60] LI J, HUO H, LI C, et al. AttentionFGAN: infrared and visible image fusion using attention-based generative adversarial networks[J]. IEEE Transactions on Multimedia, 2020, 23: 1383-1396. [61] ZHANG H, YUAN J, TIAN X, et al. GAN-FM: infrared and visible image fusion using GAN with full-scale skip connection and dual Markovian discriminators[J]. IEEE Transactions on Computational Imaging, 2021, 7: 1134-1147. [62] MA J Y, ZHANG H, SHAO Z F, et al. GANMcC: a generative adversarial network with multiclassification constraints for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-14. [63] YANG Y, LIU J X, HUANG S Y, et al. Infrared and visible image fusion via texture conditional generative adversarial network[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(12): 4771-4783. [64] 周祎楠, 杨晓敏. 基于 GAN 的红外与可见光图像融合算法[J]. 现代计算机, 2021(16): 94-97. ZHOU W N, YANG X M. Infrared and visible image fusion based on GAN[J]. Modern Computer, 2021(16): 94-97. [65] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 4700-4708. [66] LI H, WU X J. DenseFuse: a fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing, 2018, 28(5): 2614-2623. [67] 王建中, 徐浩楠, 王洪枫, 等. 基于残差密集块和自编码网络的红外与可见光图像融合[J]. 北京理工大学学报, 2021, 41(10): 1077-1083. WANG J Z, XU H N, WANG H F, et al. Infrared and visible image fusion based on residual dense block and auto-encoder network[J]. Transactions of Beijing Institute of Technology, 2021, 41(10): 1077-1083. [68] TANG L, XIANG X, ZHANG H, et al. DIVFusion: darkness-free infrared and visible image fusion[J]. Information Fusion, 2023, 91: 477-493. [69] XU H, GONG M, TIAN X, et al. CUFD: an encoder-decoder network for visible and infrared image fusion based on common and unique feature decomposition[J]. Computer Vision and Image Understanding, 2022, 218: 103407. [70] REN L, PAN Z B, CAO J Z, et al. Infrared and visible image fusion based on variational auto-encoder and infrared feature compensation[J]. Infrared Physics & Technology, 2021, 117: 103839. [71] WANG Z, WANG J, WU Y, et al. UNFusion: a unified multi-scale densely connected network for infrared and visible image fusion[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 32(6): 3360-3374. [72] WANG C, WU J, ZHU Z, et al. MSFNet: MultiStage fusion network for infrared and visible image fusion[J]. Neurocomputing, 2022, 507: 26-39. [73] JIAN L H, YANG X M, LIU Z, et al. SEDRFuse: a symmetric encoder-decoder with residual block network for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-15. [74] CHENG C, SUN C, SUN Y, et al. StyleFuse: an unsupervised network based on style loss function for infrared and visible image fusion[J]. Signal Processing: Image Communication, 2022, 106: 116722. [75] ZHAO Z, XU S, ZHANG C, et al. DIDFuse: deep image decomposition for infrared and visible image fusion[J]. arXiv:2003.09210, 2020. [76] CHENG C Y, WU X J, XU T Y, et al. UNIFusion: a lightweight unified image fusion network[J]. IEEE Transactions on Instrumentation and Measurement, 2021, 70: 1-14. [77] ZHANG G, NIE R, CAO J. SSL-WAEIE: self-supervised learning with weighted auto-encoding and information exchange for infrared and visible image fusion[J]. IEEE/CAA Journal of Automatica Sinica, 2022, 9(9): 1694-1697. [78] ZHAO F, ZHAO W, YAO L, et al. Self-supervised feature adaption for infrared and visible image fusion[J]. Information Fusion, 2021, 76: 189-203. [79] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems, 2017:?5998-6008. [80] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognition at scale[J]. arXiv:2010.11929, 2020. [81] LIU Z, LIN Y, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 10012-10022. [82] TU Z, TALEBI H, ZHANG H, et al. Maxvit: multi-axis vision transformer[C]//Proceedings of the European Conference on Computer Vision, 2022: 459-479. [83] WANG W, XIE E, LI X, et al. Pyramid vision transformer: a versatile backbone for dense prediction without convolutions[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 568-578. [84] WANG Z, CHEN Y, SHAO W, et al. SwinFuse: a residual swin transformer fusion network for infrared and visible images[J]. IEEE Transactions on Instrumentation and Measurement, 2022, 71: 1-12. [85] CHANG Z, FENG Z, YANG S, et al. AFT: adaptive fusion transformer for visible and infrared images[J]. IEEE Transactions on Image Processing, 2023, 32: 2077-2092. [86] JIA S, MIN Z, FU X. Multiscale spatial-spectral transformer network for hyperspectral and multispectral image fusion[J]. Information Fusion, 2023, 96: 117-129. [87] CHEN J, DING J, YU Y, et al. THFuse: an infrared and visible image fusion network using transformer and hybrid feature extractor[J]. Neurocomputing, 2023, 527: 71-82. [88] LIU Q, PI J, GAO P, et al. STFNet: self-supervised transformer for infrared and visible image fusion[J]. IEEE Transactions on Emerging Topics in Computational Intelligence, 2024, 8(2):?1513-1526. [89] LI H, WU X J. CrossFuse: a novel cross attention mechanism based infrared and visible image fusion approach[J]. Information Fusion, 2024, 103: 102147. [90] ZHAO W, XIE S, ZHAO F, et al. MetaFusion: infrared and visible image fusion via meta-feature embedding from object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 13955-13965. [91] VS V, VALANARASU J M J, OZA P, et al. Image fusion transformer[C]//Proceedings of the 2022 IEEE International Conference on Image Processing (ICIP), 2022: 3566-3570. [92] LIU X, GAO H, MIAO Q, et al. MFST: multi-modal feature self-adaptive transformer for infrared and visible image fusion[J]. Remote Sensing, 2022, 14(13): 3233. [93] TANG W, HE F, LIU Y, et al. DATFuse: infrared and visible image fusion via dual attention transformer[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(7):?3159-3172?. [94] TANG W, HE F, LIU Y. TCCFusion: an infrared and visible image fusion method based on transformer and cross correlation[J]. Pattern Recognition, 2023, 137: 109295. [95] ZHAO Z, BAI H, ZHANG J, et al. CDDFuse: correlation-driven dual-branch feature decomposition for multi-modality image fusion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 5906-5916. |
[1] | LI Jiayi, MA Zhiliang, CHENG Lijie, JI Xinlin. Comprehensive Review of Multimodal Data Fusion Methods for Construction Robot Localization [J]. Computer Engineering and Applications, 2024, 60(15): 11-23. |
[2] | LI Mu, HE Jincheng, YANG Heng. Fire Hazard Detection Algorithm with Dual-Branch GAN and Attention Mechanism [J]. Computer Engineering and Applications, 2024, 60(14): 228-239. |
[3] | KONG Weiwei. Medical Image Fusion Method Using SML and Side Window Filter in Framelet Transform [J]. Computer Engineering and Applications, 2024, 60(13): 237-245. |
[4] | WANG Zhen, YU Wanjun, CHEN Ying. Multi-Scale Feature Fusion Saliency Object Detection Based on RGB-D Images [J]. Computer Engineering and Applications, 2024, 60(11): 242-250. |
[5] | LIU Shuangli, HUANG Xueli, LIU Lei, XIE Yu, ZHANG Jinbao, YANG Jiangnan. Infrared and Visible Image Fusion Under Photoelectric Loads [J]. Computer Engineering and Applications, 2024, 60(1): 28-39. |
[6] | YOU Xuan’ang, ZHAO Peng, MU Xiaodong, ZHU Yongqing, SHEN Danyao. Research on Fusion of Visible and Near-Infrared Iris Based on Multi-Tasking Learning [J]. Computer Engineering and Applications, 2022, 58(21): 197-204. |
[7] | AN Xiaodong, LI Yali, WANG Fang. Overview of Infrared and Visible Image Fusion Algorithms for Automotive Driving Assistance System [J]. Computer Engineering and Applications, 2022, 58(19): 64-75. |
[8] | LI Yuchen, HUANG Yongdong. Medical Image Fusion Based on Convolutional Neural Network Model [J]. Computer Engineering and Applications, 2022, 58(15): 229-237. |
[9] | LI Yang, YANG Haitao, KONG Zhuo, ZHANG Changgong, WANG Jinyu. Review of Pixel-Level Infrared and Visible Image Fusion Methods [J]. Computer Engineering and Applications, 2022, 58(14): 40-50. |
[10] | DONG Xubin, ZHAO Qinghua. Research and Application of Improved Mask R-CNN in Aerial Image Target Detection [J]. Computer Engineering and Applications, 2021, 57(8): 133-144. |
[11] | GU Meihua, WANG Miaomiao, LI Liyao, FENG Jing. Color Image Multi-scale Fusion Graying Algorithm [J]. Computer Engineering and Applications, 2021, 57(4): 209-215. |
[12] | WANG Changcheng, ZHOU Dongming, LIU Yanyu, XIE Shidong. Multi-focus Image Fusion Algorithm Based on Unsupervised Deep Learning Model [J]. Computer Engineering and Applications, 2021, 57(21): 209-215. |
[13] | HU Yaping, KONG Weiwei, LI Meng, HUANG Cuiling. Improved Panoramic Image Mosaic Algorithm for TV Image Denoising Model [J]. Computer Engineering and Applications, 2021, 57(17): 203-209. |
[14] | LI Guang’an, CAO Yan, YUE Xiaoxin. Infrared and Visible Image Fusion Based on BEEMD Decomposition [J]. Computer Engineering and Applications, 2021, 57(14): 237-244. |
[15] | BAO Ying, WANG Leiguang, GUO Mengxiao, DAI Qinling, ZHENG Chen. Two-Step Framework for Fusing Worldview 3 Panchromatic and Shortwave Infrared Bands [J]. Computer Engineering and Applications, 2021, 57(11): 193-201. |
Viewed | ||||||
Full text |
|
|||||
Abstract |
|
|||||