光电载荷下的红外和可见光图像融合综述

doi:10.3778/j.issn.1002-8331.2305-0291

摘要/Abstract

摘要： 随着机载光电平台在军事、民用等领域的应用，红外与可见光图像的增强和融合技术逐渐成为研究的热点方向。红外设备主要依靠物体本身的热辐射进行成像，适用于恶劣自然环境和隐秘场所等特殊场景，但存在成像质量不佳、图像对比度较低、纹理细节不丰富等缺点；可见光图像的纹理细节和对比度更适合人类的视觉感知，但可见光图像在烟雾、夜间等条件下的成像效果差，不适用于隐秘的场所。因此，研究人员提出了既有可见光图像边缘、细节信息又有红外热辐射目标信息的互补图像融合方法。结合红外与可见光图像特点，对目前适用于机载平台的传统红外与可见光图像融合方法进行整理和综述，给出了现有红外与可见光融合算法中弱小目标、纹理信息缺失和计算效率问题的改进思路，并对光电载荷背景下传统红外与可见光图像融合技术未来的发展进行了展望。

关键词: 红外与可见光, 图像融合, 光电载荷

Abstract: With the application of airborne photoelectric platform in military and civil fields, the enhancement and fusion technology of infrared and visible image has gradually become a hot research direction. Infrared equipment mainly relies on the thermal radiation of the object itself for imaging, which is suitable for harsh natural environment and secret places and other special scenes, but it has some shortcomings such as poor imaging quality, low image contrast, not rich texture details and so on. The texture details and contrast of visible images are more suitable for human visual perception, but the imaging effect of visible images is poor under conditions such as smoke and night, so it is not suitable for hidden places. Therefore, the researchers proposed a complementary image fusion method with both visible image edge and detail information and infrared thermal radiation target information. Based on the characteristics of infrared and visible images, this paper summarizes the traditional infrared and visible image fusion methods applicable to airborne platforms, and gives the improvement ideas of dim small targets, texture information loss and computational efficiency in the existing infrared and visible image fusion algorithms. The future development of traditional infrared and visible image fusion technology under the background of photoelectric load is prospected.

Key words: infrared and visible light, image fusion, photoelectric load

刘爽利, 黄雪莉, 刘磊, 谢宇, 张锦宝, 杨江楠. 光电载荷下的红外和可见光图像融合综述[J]. 计算机工程与应用, 2024, 60(1): 28-39.

LIU Shuangli, HUANG Xueli, LIU Lei, XIE Yu, ZHANG Jinbao, YANG Jiangnan. Infrared and Visible Image Fusion Under Photoelectric Loads[J]. Computer Engineering and Applications, 2024, 60(1): 28-39.

参考文献

[1] 李扬, 杨海涛, 孔卓, 等.像素级红外与可见光图像融合方法综述[J].计算机工程与应用, 2022, 58(14): 40-50.
LI Y, YANG H T, KONG Z, et al. Review of pixel-level infrared and visible image fusion methods[J].Computer Engineering and Applications, 2022, 58(14): 40-50.
[2] 安晓东, 李亚丽, 王芳.汽车驾驶辅助系统红外与可见光融合算法综述[J].计算机工程与应用, 2022, 58(19): 64-75.
AN X D, LI Y L, WANG F. Overview of infrared and visible image fusion algorithms for automotive driving assistance system[J].Computer Engineering and Applications, 2022, 58(19): 64-75.
[3] 李霖, 王红梅, 李辰凯.红外与可见光图像深度学习融合方法综述[J].红外与激光工程, 2022, 51(12): 337-356.
LI L, WANG H M, LI C K.A review of deep learning fusion methods for infrared and visible images[J].Infrared and Laser Engineering, 2022, 51(12): 337-356.
[4] LING H, YAHUI H, SONG H, et al. Artificial intelligence technology in battlefield situation awareness[J].E3S Web of Conferences, 2022, 360: 1-6.
[5] 沈宏海, 黄猛, 李嘉全, 等.国外先进航空光电载荷的进展与关键技术分析[J].中国光学, 2012, 5(1): 20-29.
SHEN H H, HUANG M, LI J Q, et al. Recent progress in aerial electro-optic payloads and their key technologies[J].Chinese Optics, 2012, 5(1): 20-29.
[6] 左羽佳. 机载光电平台红外与可见光图像融合系统关键技术研究[D].长春: 中国科学院长春光学精密机械与物理研究所, 2017.
ZUO Y J. Research on key technologies of infrared and visible image fusion system of airborne photoelectric platform[D]. Changchun: Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, 2017.
[7] CHEN J, DING J, YU Y, et al. THFuse: an infrared and visible image fusion network using transformer and hybrid feature extractor[J]. Neurocomputing, 2023, 527: 71-82.
[8] PANIGRAHY C, SEAL A, GONZALO-MARTíN C, et al. Parameter adaptive unit-linking pulse coupled neural network based MRI-PET/SPECT image fusion[J]. Biomedical Signal Processing and Control, 2023, 83: 104659.
[9] KLIEWER M A, BAGLEY A R, REEDER S B, et al. Normal placental structural anatomy: ultrasound and Doppler features elucidated with US-MR image fusion and ferumoxytol-enhanced MRI[J]. Abdominal Radiology, 2023, 48(2): 744-751.
[10] SMORENBURG S P, LELY R J, SMIT-OCKELOEN I, et al. Automated image fusion during endovascular aneurysm repair: a feasibility and accuracy study[J]. International Journal of Computer Assisted Radiology and Surgery, 2023, 2023: 1-9.
[11] FENG X, FANG C, QIU G, et al. Multimodal medical image fusion based on visual saliency map and multichannel dynamic threshold neural P systems in sub-window variance filter domain[J]. Biomedical Signal Processing and Control, 2023, 84: 104794.
[12] LI J, HU H, LEI L, et al. Remote sensing image fusion method based on progressive cascaded deep residual network[J]. Wireless Communications and Mobile Computing, 2023, 2023: 1-9.
[13] FANG A, WU J, LI Y, et al. Infrared and visible image fusion via mutual information maximization[J]. Computer Vision Image Understanding, 2023, 231(2023): 103683.
[14] WU J, SHEN T, WANG Q, et al. Local adaptive illumination-driven input-level fusion for infrared and visible object detection[J].Remote Sensing, 2023, 15(3): 1-19.
[15] FENDRI E, BOUKHRISS R R, HAMMAMI M. Fusion of thermal infrared and visible spectra for robust moving object detection[J]. Pattern Analysis Applications, 2017, 20(4): 907-926.
[16] 张靓.一种战场红外与可见光灰度侦察图像融合方法[J].舰船电子工程, 2017, 37(12): 76-80.
ZHANG L. Battlefied reconnaissane infrared and visible gray images fusion[J].Ship Electronic Engineering, 2017, 37(12): 76-80.
[17] 吴楠, 穆成坡, 贺杨, 等.基于嵌套连接的多尺度红外与可见光图像融合[J/OL].北京航空航天大学学报: 1-11[2023-04-26]. https: //doi.org/10.13700/j.bh.1001-5965.2023.0077.
WU N, MU C P, HE Y, et al. Multi-scale infrared and visible image fusion based on nest connections[J/OL].Journal of Beijing University of Aeronautics and Astronautics: 1-11[2023-04-26]. https://doi.org/10.13700/j.bh.1001-5965.2023.0077.
[18] 姜迈, 郑岩.基于SURF-HOG与显著性特征的红外可见光图像配准融合[J].激光与红外, 2023, 53(2): 261-270.
JIANG M, ZHENG Y. Infrared and visible image registration and fusion based on SURF-HOG and saliency feature[J].Laser and Infrared, 2023, 53(2): 261-270.
[19] LI G, LIN Y, QU X. An infrared and visible image fusion method based on multi-scale transformation and norm optimization[J]. Information Fusion, 2021, 71: 109-129.
[20] LI S, ZOU Y, WANG G, et al. Infrared and visible image fusion method based on a principal component analysis network and image pyramid[J]. Remote Sensing, 2023, 15(3): 1-22.
[21] MA J, YU W, LIANG P, et al. FusionGAN: a generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48: 11-26.
[22] ZHANG B, GAO M, CHEN P, et al. Infrared and visible image fusion based on contrast enhancement guided filter and infrared feature decomposition[J]. Infrared Physics Technology, 2022, 127: 104404.
[23] XING C, WANG Z, OUYANG Q, et al. Method based on bitonic filtering decomposition and sparse representation for fusion of infrared and visible images[J]. IET Image Processing, 2018, 12(12): 2300-2310.
[24] OUJEBBOUR F Z, DOUDET V K. Image fusion based on evidence theory for multi-energy X-Ray computed tomography[J]. Journal of Nondestructive Evaluation, 2022, 41(3): 53.
[25] TOET A, HOGERVORST M A. Progress in color night vision[J]. Optical Engineering, 2012, 51(1): 010901.
[26] LIU J, FAN X, HUANG Z, et al. Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 5802-5811.
[27] JIA X, ZHU C, LI M, et al. LLVIP: a visible-infrared paired dataset for low-light vision[C]//Proceedings of the IEEE/CVF international Conference on Computer Vision, 2021: 3496-3504.
[28] LI C, ZHAO N, LU Y, et al. Weighted sparse representation regularized graph learning for RGB-T object tracking[C]//Proceedings of the 25th ACM International Conference on Multimedia, 2017: 1856-1864.
[29] ZHANG M M, CHOI J, DANIILIDIS K, et al. VAIS: a dataset for recognizing maritime imagery in the visible and infrared spectrums[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2015.
[30] HWANG S, PARK J, KIM N, et al. Multispectral pedestrian detection: benchmark dataset and baseline[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 1037-1045.
[31] LIU X, AI Y, TIAN B, et al. Robust and fast registration of infrared and visible images for electro-optical pod[J]. IEEE Transactions on Industrial Electronics, 2018, 66(2): 1335-1344.
[32] ZHANG X, LIU G, HUANG L, et al. IVOMFuse: an image fusion method based on infrared-to-visible object mapping[J]. Digital Signal Processing, 2023, 137: 104032.
[33] TANG L, YUAN J, MA J. Image fusion in the loop of high-level vision tasks: a semantic-aware real-time infrared and visible image fusion network[J]. Information Fusion, 2022, 82: 28-42.
[34] LIU J, HAN C Z, YAO X H, et al. Evidence theory-based mixture particle filter for joint detection and tracking of multiple targets[J]. IET Radar, Sonar & Navigation, 2012, 6(7): 649-658.
[35] FAN X, SHI P, NI J, et al. A thermal infrared and visible images fusion based approach for multitarget detection under complex environment[J]. Mathematical Problems in Engineering, 2015, 2015: 1-11.
[36] 赵烨. 红外与可见光图像融合的无人机目标检测方法研究[D].哈尔滨：哈尔滨工业大学, 2019.
ZHAO Y. Research on target detection method of UVA based on infrared and visible images[D]. Harbin：Harbin Institute of Technology, 2019.
[37] SUN J, ZHU H, XU Z, et al. Poisson image fusion based on Markov random field fusion model[J]. Information Fusion, 2013, 14(3): 241-254.
[38] GADE R, MOESLUND T B. Thermal cameras and applications: a survey[J]. Machine Vision Applications, 2014, 25: 245-262.
[39] 李燕苹. 复杂背景下基于多传感器图像融合的远距离小目标检测[D].深圳：深圳大学, 2007.
LI Y P. Long range small target detection based on multi sensor image fusion in complex background[D]. Shenzhen: Shenzhen University, 2007.
[40] MAHALAXMI U S B K, RAMA RAJU N N S V, RAMESH PATNAIK M. Hybrid fusion approach of medical images using DWT, DCT, Laplacian pyramid and curvelet transform[J]. Journal of Critical Reviews, 2020, 7(14): 601-609.
[41] XU Q. Image fusion and stylization processing based on multiscale transformation and convolutional neural network[J]. Computational Intelligence Neuroscience, 2022, 2022: 1181189.
[42] 王长龙, 刘贺, 张帅, 等.基于NSDTCT与稀疏表示的红外和微光图像融合[J].陆军工程大学学报, 2022, 1(4): 8-13.
WANG C L, LIU H, ZHANG S, et al. Fusion of infrared low-light-level images based on NSDTCT and sparse representation[J].Journal of Army Engineering University of PLA, 2022, 1(4): 8-13.
[43] JIN H, JIAO L, LIU F, et al. Fusion of infrared and visual images based on contrast pyramid directional filter banks using clonal selection optimizing[J]. Optical Engineering, 2008, 47(2): 027002.
[44] CHANDANA K, REDDY V U, RAMASHRI T. A novel image fusion technique using dual tree complex wavelet transform based Laplacian pyramidal decomposition[J]. International Journal of Engineering Research Technology, 2015, 11: 227-232.
[45] 曾祥通, 张玉珍, 孙佳嵩.颜色对比度增强的红外与可见光图像融合方法[J].红外与激光工程, 2015, 44(4): 1198-1202.
ZENG X T, ZHANG Y Z, SUN J S. One color contrast enhanced infrared and visible image fusion method.[J].Infrared and Laser Engineering, 2015, 44(4): 1198-1202.
[46] HAN X, ZHANG L L, DU L Y, et al. Fusion of infrared and visible images based on discrete wavelet transform[C]//Proceedings of the Second Symposium on New Detection Technology and Its Application, National Defense Optoelectronics Forum, 2015.
[47] BAVIRISETTI D P, DHULI R. Two-scale image fusion of visible and infrared images using saliency detection[J]. Infrared Physics Technology, 2016, 76: 52-64.
[48] NAIDU A R, BHAVANA D, REVANTH P, et al. Fusion of visible and infrared images via saliency detection using two-scale image decomposition[J]. International Journal of Speech Technology, 2020, 23(4): 815-824.
[49] SUN H, LIU Q, WANG J, et al. Fusion of infrared and visible images for remote detection of low-altitude slow-speed small targets[J]. IEEE Journal of Selected Topics in Applied Earth Observations Remote Sensing, 2021, 14: 2971-2983.
[50] LI Y, LIU G, BAVIRISETTI D P, et al. Infrared-visible image fusion method based on sparse and prior joint saliency detection and LatLRR-FPDE[J]. Digital Signal Processing, 2023, 134: 103910.
[51] WEI H, ZHU Z, CHANG L, et al. A novel precise decomposition method for infrared and visible image fusion[C]//Proceedings of the 38th Chinese Control Conference.Shanghai: Shanghai Systems Science Press, 2019: 3341-3345.
[52] REN L, PAN Z, CAO J, et al. Infrared and visible image fusion based on weighted variance guided filter and image contrast enhancement[J]. Infrared Physics and Technology, 2021, 114: 103662.
[53] ZHANG X, MA Y, FAN F, et al. Infrared and visible image fusion via saliency analysis and local edge-preserving multi-scale decomposition[J].JOSA A, 2017, 34(8): 1400-1410.
[54] LIU Y, LIU S, WANG Z. A general framework for image fusion based on multi-scale transform and sparse representation[J]. Information Fusion, 2015, 24: 147-164.
[55] MA J, ZHOU Z, WANG B, et al. Infrared and visible image fusion based on visual saliency map and weighted least square optimization[J]. Infrared Physics Technology, 2017, 82: 8-17.
[56] LIN Y, CAO D. Adaptive infrared and visible image fusion method by using rolling guidance filter and saliency detection[J]. Optik, 2022, 262: 169218.
[57] LIU Y, DONG L, XU W. Infrared and visible image fusion for shipborne electro-optical pod in maritime environment[J]. Infrared Physics Technology, 2023, 128: 104526.
[58] 肖旭光, 肖刚, 敬忠良.空间目标融合识别算法及实验研究[J].计算机工程与应用, 2011, 47(8): 154-156.
XIAO X G, XIAO G, JING Z L. Study on space target recognition algorithm and its experiment[J].Computer Engineering and Applications, 2011, 47(8): 154-156.
[59] 白玉, 侯志强, 刘晓义, 等.基于可见光图像和红外图像决策级融合的目标检测算法[J].空军工程大学学报 (自然科学版), 2020, 21(6): 53-59.
BAI Y, HOU Z Q, LIU X Y, et al. An object detection algorithm based on decision-level fusion of visible light image and infrared images[J].Journal of Air Force Engineering University, 2020, 21(6): 53-59.
[60] 朱铠铠. 机载多源融合目标智能检测技术研究[D].西安：西安电子科技大学, 2022.
ZHU K K. Study on intelligent detection technology for multi-source fusion target based on airborne platform[D].Xi’an: Xidian University, 2022.
[61] 刘莲. 基于决策级融合的无人机目标跟踪方法研究[D].成都: 电子科技大学, 2022.
LIU L. Research on UAV target tracking method based on decision level fusion[D].Chengdu: University of Electronic Science and Technology, 2022.
[62] 宁大海, 郑晟.可见光和红外图像决策级融合目标检测算法[J].红外技术, 2023, 45(3): 282-291.
NING D H, ZHENG S. An object detection algorithm based on decision-level fusion of visible light image and infrared images[J].Infrared Technology, 2023, 45(3): 282-291.
[63] 唐聪, 凌永顺, 杨华, 等.基于深度学习的红外与可见光决策级融合跟踪[J].激光与光电子学进展, 2019, 56(7): 217-224.
TANG C, LING Y S, YANG H, et al. Decision-level fusion tracking for Infrared and visible spectra based on deep learning[J]. Laser & Optoelectronics Progress, 2019, 56(7): 217-224.
[64] HU Z H, JING Y G, WU G Q. Decision-level fusion detection method of visible and infrared images under low light conditions[J]. EURASIP Journal on Advances in Signal Processing, 2023, 2023(1): 1-13.
[65] RUDIN L I, OSHER S, FATEMI E. Nonlinear total variation based noise removal algorithms[J]. Physica D: Nonlinear Phenomena, 1992, 60: 259-268.
[66] MA J, CHEN C, LI C, et al. Infrared and visible image fusion via gradient transfer and total variation minimization[J]. Information Fusion, 2016, 31: 100-109.
[67] GUO H, MA Y, MEI X, et al. Infrared and visible image fusion based on total variation and augmented Lagrangian[J]. JOSA A, 2017, 34(11): 1961-1968.
[68] NIE R, MA C, CAO J, et al. A total variation with joint norms for infrared and visible image fusion[J]. IEEE Transactions on Multimedia, 2021, 24: 1460-1472.
[69] DU Q, XU H, MA Y, et al. Fusing infrared and visible images of different resolutions via total variation model[J]. Sensors, 2018, 18(11): 1-17.
[70] ZHAO J, CUI G, GONG X, et al. Fusion of visible and infrared images using global entropy and gradient constrained regularization[J]. Infrared Physics Technology, 2017, 81: 201-209.
[71] ZHANG X, DAI X, ZHANG X, et al. Joint principal component analysis and total variation for infrared and visible image fusion[J]. Infrared Physics Technology, 2023, 128: 104523.
[72] MA J, MA Y, LI C. Infrared and visible image fusion methods and applications: a survey[J]. Information Fusion, 2019, 45: 153-178.
[73] LUO X, ZHANG Z, WU X. A novel algorithm of remote sensing image fusion based on shift-invariant Shearlet transform and regional selection[J]. AEU-International Journal of Electronics Communications, 2016, 70(2): 186-197.
[74] ZHANG Z, CUI J, LUO X, et al. Statistical correlative model in the multimodal fusion of brain images[J]. International Journal of Imaging Systems Technology, 2020, 30(4): 1066-1079.
[75] LUO X, ZHANG Z, ZHANG B, et al. Image fusion with contextual statistical similarity and nonsubsampled shearlet transform[J]. IEEE Sensors Journal, 2016, 17(6): 1760-1771.
[76] LUO X, JIANG Y, WANG A, et al. Infrared and visible image fusion based on multi-state contextual hidden Markov model[J]. Pattern Recognition, 2023, 138: 109431.
[77] YUN Z, MUBARAK S. Visual attention detection in video sequences using spatiotemporal cues[C]//Proceedings of the 14th ACM International Conference on? Multimedia, 2006: 815-824.
[78] ZOU D, YANG B. Infrared and low-light visible image fusion based on hybrid multiscale decomposition and adaptive light adjustment[J]. Optics Lasers in Engineering, 2023, 160: 107268.
[79] MA Z, YUAN X, HAN S, et al. Improved chaotic particle swarm optimization algorithm with more symmetric distribution for numerical function optimization[J]. Symmetry, 2019, 11(7): 1-19.
[80] TAWHID M A, ALI A F. Simplex particle swarm optimization with arithmetical crossover for solving global optimization problems[J]. Opsearch, 2016, 53: 705-740.
[81] ZHANG J, TANG B, HU S. Infrared and visible image fusion based on particle swarm optimization and dense block[J]. Frontiers in Energy Research, 2022, 10: 1001450.
[82] LI H F, ZHAO J Z, YUN J X, et al. Feature dynamic alignment and refinement for infrared-visible image fusion: translation robust fusion[J]. Information Fusion, 2023, 95: 26-41.
[83] DONG L L, WANG J, ZHAO L J. FusionJISI: a fusion algorithm based on infrared and visible images with joint involvement of source image[J]. Infrared Physics and Technology, 2023, 132: 104704.
[84] GOSHTASBY A, NIKOLOV S G. Image fusion: advances in the state of the art[J]. Information Fusion, 2006, 8(2): 114-118.
[85] AYDIN F, UGURDAG H F, LEVENT V E, et al. Rapid design of real-time image fusion on FPGA using HLS and other techniques[C]//2018 IEEE/ACS 15th International Conference on Computer Systems and Applications (AICCSA), 2018: 1-6.
[86] ROCKINGER O. Image sequence fusion using a shift-invariant wavelet transform[C]//Proceedings of International Conference on Image Processing, 1997: 288-291.
[87] SIMS O, IRVINE J. An FPGA implementation of pattern-selective pyramidal image fusion[C]//2006 International Conference on Field Programmable Logic and Applications, 2006: 1-4.
[88] SONG Y J, GAO K, NI G Q, et al. Implementation of real-time Laplacian pyramid image fusion processing based on FPGA[J]. Beijing Institute of Technology (China), 2007, 6833: 315-322.
[89] LI H, LIU L, HUANG W, et al. An improved fusion algorithm for infrared and visible images based on multi-scale transform[J]. Infrared Physics Technology, 2016, 74: 28-37.
[90] YAN C M, GUO B L, YI M. Fast algorithm for nonsubsampled contourlet transform[J]. Acta Automatica Sinica, 2014, 40(4): 757-762.
[91] YAN L, YU Z, HAN N, et al. Improved image fusion algorithm for detecting obstacles in forests[J]. J Digit Inf Manag, 2013, 11(5): 378-384.
[92] MISHRA A, MAHAPATRA S, BANERJEE S. Modified Frei-Chen operator-based infrared and visible sensor image fusion for real-time applications[J]. IEEE Sensors Journal, 2017, 17(14): 4639-4646.
[93] JEONG W, HAN B G, YANG H S, et al. Real-time visible-infrared image fusion using multi-guided filter[J]. KSII Transactions on Internet Information Systems, 2019, 13(6): 3092-3107.
[94] AWAD M, ELLIETHY A, ALY H A. Real-time visible and near-infrared video fusion: architecture and implementation[J]. Journal of Real-Time Image Processing, 2021, 18(6): 2479-2493.
[95] ZHANG L, YANG X, WAN Z, et al. A real-time FPGA implementation of infrared and visible image fusion using guided filter and saliency detection[J]. Sensors, 2022, 22(21): 1-18.
[96] LI J, PENG Y, JIANG T. Embedded real-time infrared and visible image fusion for UAV surveillance[J] Journal of Real-Time Image Processing, 2021, 18(6): 2331-2345.
[97] 黄西莹, 王洁, 王娇颖, 等.多源图像融合算法实时优化方法[J].应用光学, 2022, 43(4): 676-681.
HUANG X Y, WANG J, WANG J Y, et al. Real-time implementation of multi-source image fusion algorithm[J]. Applied Optics, 2022, 43(4): 676-681.