Super-Resolution Reconstruction of Aerial Images Based on Hierarchical Feature Fusion Network

doi:10.3778/j.issn.1002-8331.2101-0370

Abstract

Abstract: Most of the current image super-resolution network models based on deep learning have high model complexity and low feature utilization, which leading to poor performance of final reconstruction due to severe feature loss, especially when they are applied in a complex shooting environment. In order to solve the problems mentioned above, a hierarchical feature fusion network（HFFN） for image super-resolution is proposed. Firstly, a symmetrical layered structure is introduced to enhance the fusion of image features at different levels. Secondly, it can not only reduce local residual loss but the gradient vanishing and gradient exploding can be effectively avoided by using a denser residual connected structure. Finally, by adding attention module to each residual block, the sensitivity of the model to high-frequency information of the image is enhanced. The proposed model has been used in the super-resolution reconstruction of high-altitude aerial photography environment for verifying its effectiveness. The experimental results show that the proposed algorithm has remarkable reconstruction performance and its average PSNR is increased by 0.31?dB in comparison with the EDSR algorithm when tested in 14 different aerial environments, especially the reconstruction of complex scenes.

Key words: super-resolution reconstruction, dense residual structure, channel attention, aerial image reconstruction

摘要： 基于深度学习的图像超分辨率网络模型复杂度高，特征利用率较低，尤其是应用在复杂拍摄环境中的图像超分辨率重建，由于特征损失严重，最终重建的效果也较差。针对以上问题，提出分层特征融合图像超分辨率网络。引入对称式的分层结构，以增强不同层次图像特征的融合；使用更为密集的残差连接结构，减少局部残差损失，同时缓解梯度消失和梯度爆炸问题；在每个残差块中加入注意力机制，增强网络对图像高频信息的敏感度。为了验证算法在复杂环境中的效果，将模型应用于高空航拍图像超分辨率重建中。实验结果表明，所提算法相比于EDSR算法，在14个不同航拍图像环境中，尤其是复杂场景下的重建，平均PSNR提高了0.31?dB，效果显著。

关键词: 图像超分辨率重建, 密集残差结构, 通道注意力, 航拍图像重建

YANG Xianing, WANG Banghai, LI Zuolong. Super-Resolution Reconstruction of Aerial Images Based on Hierarchical Feature Fusion Network[J]. Computer Engineering and Applications, 2022, 58(19): 224-232.

杨夏宁, 王帮海, 李佐龙. 分层特征融合网络航拍图像超分辨率重建[J]. 计算机工程与应用, 2022, 58(19): 224-232.

References

[1] 蒋文杰，罗晓曙，戴沁璇.基于对抗网络人脸超分辨率重建算法研究[J].计算机工程与应用，2021，57（11）：219-223.
JIANG W J，LUO X S，DAI Q X.Research on face super-resolution reconstruction algorithm based on gene-rative adversarial networks[J].Computer Engineering and Applications，2021，57（11）：219-223.
[2] 卢峰，周琳，蔡小辉.面向安防监控场景的低分辨率人脸识别算法研究[J].计算机应用研究，2021，38（4）：1230-1234.
LU F，ZHOU L，CAI X H.Research on low resolution face recognition algorithm for security surveillance scene[J].Application Research of Computers，2021，38（4）：1230-1234.
[3] 蒋文杰，罗晓曙，戴沁璇.基于对抗网络遥感图像超分辨率重建研究[J].计算机工程与应用，2020，56（21）：199-203.
JIANG W J，LUO X S，DAI Q X.Research on super- resolution reconstruction of remote sensing image based on improved conditional generative adversarial networks[J].Computer Engineering and Applications，2020，56（21）：199-203.
[4] CHEN Y，CHRISTODOULOU A G，ZHOU Z，et al.MRI super-resolution with GAN and 3D multi-level DenseNet：Smaller，faster，and better[J].arXiv：2003.01217，2020.
[5] 谢海平，谢凯利，杨海涛.图像超分辨率方法研究进展[J].计算机工程与应用，2020，56（19）：34-41.
XIE H P，XIE K L，YANG H T.Research progress of image super-resolution methods[J].Computer Engineering and Applications，2020，56（19）：34-41.
[6] TOMASI C，MANDUCHI R.Bilateral filtering for gray and color images[C]//Proceedings of the IEEE International Conference on Computer Vision，1998：839-846.
[7] LIU C，SUN D Q.On Bayesian adaptive video super resolution[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2014，36（2）：346-360.
[8] 刘钢，彭群生，鲍虎军.基于图像建模技术研究综述与展望[J].计算机辅助设计与图形学学报，2005，17（1）：18-27.
LIU G，PENG Q S，BAO H J.Review and prospect of image-based modeling techniques[J].Journal of Computer-aided Design and Computer Graphics，2005，17（1）：18-27.
[9] DONG C，LOY C C，HE K，et al.Learning a deep convolutional network for image super-resolution[C]//Proceedings of the European Conference on Computer Vision，2014：184-199.
[10] DONG C，LOY C C，HE K，et al.Image super-resolution using deep convolutional networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2015，38（2）：295-307.
[11] DONG C，LOY C C，TANG X O.Accelerating the super-resolution convolutional neural network[C]//Proceedings of the European Conference on Computer Vision，2016：391-407.
[12] KIM J，KWON LEE J，MU LEE K.Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：1646-1654.
[13] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[14] KIM J，KWON LEE J，MU LEE K.Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Las Vegas，Jun 27-30，2016：1637-1645.
[15] ZAREMBA W，SUTSKEVER I，VINYALS O.Recurrent neural network regularization[J].arXiv：1409.2329，2014.
[16] TAI Y，YANG J，LIU X M，Image super-resolution via deep recursive residual network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Honolulu，Jul 21-26，2017：2790-2798.
[17] TONG T，LI G，LIU X，et al.Image super-resolution using dense skip connections[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：4799-4807.
[18] HUANG G，LIU Z，VAN DER MAATEN L，et al.Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：4700-4708.
[19] LIM B，SON S，KIM H，et al.Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：136-144.
[20] RONNEBERGER O，FISCHER P，BROX T.U-Net：Convolutional networks for biomedical image segmentation[C]//Proceedings of the Medical Image Computing and Computer-Assisted Intervention，2015：234-241.
[21] ZEILER M，KRISHNAN D，TAYLOR G，et al.Deconvolutional networks[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2010：2528-2535.
[22] HU J，SHEN L，ALBANIE S，et al.Squeeze-and-excitation networks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2020，42（8）：2011-2023.
[23] ZHANG Y L，LI K P，LI K，et al.Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the European Conference on Computer Vision，2018：294-310.
[24] LIU J，ZHANG W，TANG Y，et al.Residual feature aggregation network for image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，Seattle，Jun 13-19，2020：2356-2365.
[25] WANG Q，WU B G，ZHU P F，et al.ECA-Net：Efficient channel attention for deep convolutional neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2020：11531-11539.
[26] SHI W，CABALLERO J，HUSZAR F，et al.Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：1874-1883.
[27] XIA G S，BAI X，DING J，et al.DATA：A large-scale dataset for object detection in aerial images[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：3974-3983.
[28] WANG X，YU K，WU S X，et al.ESRGAN：Enhanced super-resolution generative adversarial networks[C]//Proceedings of the European Conference on Computer Vision Workshops，2018.