Multi-Coupled Feedback Networks for Image Fusion and Super-Resolution Methods

doi:10.3778/j.issn.1002-8331.2212-0118

Abstract

Abstract: People often need to obtain high dynamic range and high resolution images in their daily life. However, due to the limitation of technical equipment, high dynamic range images are often obtained by multi-exposure fusion (MEF) of low dynamic range images, and high resolution images are often obtained by super resolution (SR) of low resolution images. MEF and SR are usually studied as two separate elements. In order to solve the problem that the current model cannot achieve high dynamic range and high resolution at the same time, a multi-coupling feedback network (MCF-Net) and its method are proposed in this paper through the study of existing methods. The model includes：[N] subnets and output modules; in the method, first, [N] downsampled images [Iilr,Imlr,I-ilr] are input to [N] subnets respectively, and the extracted low-resolution features [Filr,Fmlr,F-ilr]; then the super-resolution features [Gi0,Gm0,G-i0] of the corresponding images are extracted according to the low-resolution features; the fused high-resolution features [Git,Gmt,G-it] are obtained and input to the next MCFB until the T-th MCFB obtains the fused high-resolution features [GiT,GmT,G-iT]; then the corresponding fused super-resolution image [Iit,Imt,I-it] is obtained; finally the high dynamic range, super-resolution image [Iout] is obtained by fusing the output [IiT,ImT,I-iT] of the [T]-th reconstruction module REC in [N] subnets. In this paper, the performance is experimented and verified on the SICE dataset, and compared with 33 existing methods, the results show that each of the following evaluation indexes has been significantly improved, including the structural similarity (SSIM) reaching 0.833 2, the peak signal-to-noise ratio (PSNR) reaching 22.07 dB, and the multi-exposure fusion similarity (MEF-SSIM) reaching 0.937 8.

Key words: image multi-exposure fusion, image super-resolution, convolutional neural network, computer vision, deep learning

摘要： 人们在日常生活中往往需要得到高动态范围和高分辨率的图像。但由于技术设备的限制，高动态范围的图像往往通过低动态范围图像的多曝光融合（MEF）而获得，高分辨率图像往往通过低分辨率图像的超分辨率（SR）而获得。MEF和SR通常被作为两个独立的内容进行研究。为了解决当前模型不能同时实现高动态范围和高分辨率的问题，通过对现有方法进行研究，提出了一种基于多耦合反馈网络MCF-Net及其方法。模型包括：[N]个子网和输出模块；在方法中，将[N]张下采样图片[Iilr,Imlr,I-ilr]分别输入至[N]个子网，提取的低分辨率特征[Filr,Fmlr,F-ilr]；根据低分辨率特征[Filr,Fmlr,F-ilr]提取对应图像的超分辨率特征[Gi0,Gm0,G-i0]；得到融合高分辨率特征[Git,Gmt,G-it]并输入至下个MCFB中，直至第[T]个MCFB得到融合高分辨率特征[GiT,GmT,G-iT]；获取对应的融合超分辨率图像[Iit,Imt,I-it]；融合[N]个子网中第[T]个重建模块REC输出的[IiT,ImT,I-iT]得到高动态范围、超分辨率图像[Iout]。在SICE数据集上实验并验证了性能，与现有的33种方法进行对比，结果显示以下各评价指标都有明显的提高，其中结构相似性（SSIM）达到0.833 2，峰值信噪比（PSNR）达到22.07 dB，多曝光融合相似性（MEF-SSIM）达到0.937 8。

关键词: 图像多曝光融合, 图像超分辨率, 卷积神经网络, 计算机视觉, 深度学习

WANG Rong, DUANMU Chunjiang. Multi-Coupled Feedback Networks for Image Fusion and Super-Resolution Methods[J]. Computer Engineering and Applications, 2024, 60(5): 210-220.

王蓉, 端木春江. 多耦合反馈网络的图像融合和超分辨率方法[J]. 计算机工程与应用, 2024, 60(5): 210-220.

References

[1] MA K D, DUANMU Z F, ZHU H W, et al. Deep guided learning for fast multi-exposure image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 2808-2819.
[2] BAVIRISETTI D P, XIAO G, ZHAO J H, et al. Multi-scale guided image and video fusion: a fast and efficient approach[J]. Circuits Systems and Signal Processing, 2019, 38(12): 5576-5605.
[3] MA K, LI H, YONG H W, et al. Robust multi-exposure image fusion: a structural patch decomposition approach[J]. IEEE Transactions on Image Processing, 2017, 26(5): 2519-2532.
[4] LI H, MA K D, YONG H W, et al. Fast multi-scale structural patch decomposition for multi-exposure image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 5805-5816.
[5] XU H, MA J Y, ZHANG X P. MEF-GAN: multi-exposure image fusion via generative adversarial networks[J]. IEEE Transactions on Image Processing, 2020, 29: 7203-7216.
[6] DONG C, LOY C C, HE K, et al. Learning a deep convolutional network for image super-resolution[C]//European Conference on Computer Vision, 2014: 184-199.
[7] KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1646-1654.
[8] DONG C, LOY C C, TANG X. Accelerating the super-resolution convolutional neural network[C]//European Conference on Computer Vision, 2016: 391-407.
[9] HARIS M, SHAKHNAROVICH G, UKITA N. Deep back-projection networks for single image super-resolution[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(12): 4323-4337.
[10] LI Z, YANG J L, LIU Z, et al. Feedback network for image super-resolution[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 3867-3876.
[11] DENG X, ZHANG Y T, XU M, et al. Deep coupled feedback network for joint exposure fusion and image super-resolution[J]. IEEE Transactions on Image Processing, 2021, 30: 3098-3112.
[12] CAI J R, GU S H, ZHANG L. Learning a deep single image contrast enhancer from multi-exposure images[J]. IEEE Transactions on Image Processing, 2018, 27(4): 2049-2062.
[13] KONG F Y, LI M X, LIU S W, et al. Residual local feature network for efficient super-resolution[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2022: 765-775.
[14] ZOU W B, TIAN Y, ZHENG W X, et al. Self-calibrated efficient transformer for lightweight super-resolution[C]//IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2022: 929-938.
[15] ZHANG T L, LI K P, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]//European Conference on Computer Vision, 2018: 294-310.
[16] LIANG J Y, CAO J Z, SUN G L, et al. SwinIR: image restoration using swin transformer[C]//IEEE/CVF International Conference on Computer Vision Workshops, 2021.
[17] XU H, MA J Y, JIANG J J, et al. U2Fusion: a unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 502-518.
[18] WANG Z, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[19] MA K, ZENG K, WANG Z. Perceptual quality assessment for multi-exposure image fusion[J]. IEEE Transaction on Image Processing, 2015, 24(11): 3345-3356.