面向大气湍流畸变场景的卷积图像校正方法

doi:10.3778/j.issn.1002-8331.2309-0177

摘要/Abstract

摘要： 大气湍流引发的折射率在时间和空间上的随机波动，会使远程成像系统捕获的图像出现时空模糊和几何畸变，严重削弱了图像的视觉效果和应用价值。针对这一问题，许多学者尝试采用多帧幸运区的方法和基于卷积神经网络的深度学习方法来修复大气湍流引起的图像畸变。然而，在强湍流情况下，这些方法通常模型训练难度较大，湍流自适应性校正能力较为薄弱。为了解决上述问题，经研究提出了一种改进的深度学习网络来提升湍流畸变校正的性能。该网络采用Transformer端到端的网络结构，利用多头自注意力跨通道捕获局部上下文信息；对一级校正网络采用蒙特卡洛Dropout策略进行训练，通过模型不确定性来提取传统方法难以捕获的退化区域；利用提取到的不确定性映射作为引导信息，输入第二级校正网络提升校正准确性。在基于时空模糊加几何畸变的合成湍流退化图像集上进行了实验，证明了提出方法的有效性。

关键词: 图像处理, 大气湍流, 自注意力机制, 不确定性映射

Abstract: The random fluctuations in refractive index caused by atmospheric turbulence in time and space can cause spatiotemporal blur and geometric distortion in images captured by remote imaging systems, seriously weakening the visual effect and application value of the images. In response to this issue, many scholars have attempted to use multi frame lucky zone methods and deep learning methods based on convolutional neural networks to repair image distortion caused by atmospheric turbulence. However, in the case of strong turbulence, these methods usually have greater difficulty in model training and weaker adaptive correction ability for turbulence. To address the above issues, an improved deep learning network has been proposed through research to enhance the performance of turbulence distortion correction. The network adopts a Transformer end-to-end network structure, utilizing multi head self attention to capture local contextual information across channels. At the same time, the first level correction network is trained using a Monte Carlo Dropout strategy to extract degraded regions that are difficult to capture by traditional methods through model uncertainty. Finally, using the extracted uncertainty map as guidance information, input it into the second level correction network to improve the accuracy of correction. Experiments are conducted on a synthetic turbulence degraded image set based on spatiotemporal blur and geometric distortion, demonstrating the effectiveness of the proposed method.

Key words: image processing, atmospheric turbulence, self attention mechanism, uncertainty map

成宽洪, 吴钰博, 朱凌建, 李佳, 李军怀. 面向大气湍流畸变场景的卷积图像校正方法[J]. 计算机工程与应用, 2024, 60(11): 215-223.

CHENG Kuanhong, WU Yubo, ZHU Lingjian, LI Jia, LI Junhuai. Convolutional Image Correction Model for Atmospheric Turbulence Distortion[J]. Computer Engineering and Applications, 2024, 60(11): 215-223.

参考文献

[1] 刘超, 陈善球, 廖周, 等.自适应光学技术在通信波段对大气湍流的校正[J]. 光学精密工程, 2014, 22(10): 2605-2610.
LIU C, CHEN S Q, LIAO Z, et al. Correction of atmospheric turbulence by adaptive optics in waveband of free-space coherent laser communication[J]. Editorial Office of Optics and Precision Engineering, 2014, 22(10): 2605-2610.
[2] 王建立, 董玉磊, 姚凯男, 等. 349单元自适应光学波前处理器[J]. 光学精密工程, 2018, 26(5): 1007-1013.
WANG J L, DONG Y L, YAO K N, et al. Three hundred and fourty-nine unit adaptive optical wavefront processor[J]. Optics and Precision Engineering, 2018, 26(5): 1007-1013.
[3] 刘超, 胡立发, 穆全全, 等. 校正水平湍流波面的自适应光学系统的带宽需求[J]. 光学精密工程, 2010, 18(10): 2137-2142.
LIU C, HU L F, MU Q Q, et al. Bandwidth requirements of adaptive optical system for horizontal turbulence correction[J]. Editorial Office of Optic sand Precision Engineering, 2010, 18(10): 2137-2142.
[4] 刘万军, 张正寰, 曲海成. 融合DenseNet的多尺度图像去模糊模型[J]. 计算机工程与应用, 2021, 57(24): 219-226.
LIU W J, ZHANG Z H, QU H C. Multi-scale image deblurring model with DenseNet[J]. Computer Engineering and Applications, 2021, 57(24): 219-226.
[5] ZHU X, MILANFAR P. Removing atmospheric turbulence via space-invariant deconvolution[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 35(1): 157-170.
[6] 李灏, 杨志景, 王美林, 等. 噪声和U型判别网络的真实世界图像超分辨率[J]. 计算机工程与应用, 2023, 59(6): 204-211.
LI H, YANG Z J, WANG M L, et al. Real-world image super-resolutioin based on noise and U-shape discrimination network[J]. Computer Engineering and Applications, 2023, 59(6): 204-211.
[7] 黄健, 赵元元, 郭苹, 等. 深度学习的单幅图像超分辨率重建方法综述[J]. 计算机工程与应用, 2021, 57(18): 13-23.
HUANG J, ZHAO Y Y, GUO P, et al. Survey of single image super-resolution based on deep learning[J]. Computer Engineering and Applications, 2021, 57(18): 13-23.
[8] ANANTRASIRICHAI N, ACHIM A, KINGSBURY N G, et al. Atmospheric turbulence mitigation using complex wavelet-based fusion[J]. IEEE Transactions on Image Processing, 2013, 22(6): 2398-2408.
[9] MAO Z Y, CHIMITT N, CHAN S H. Accelerating atmospheric turbulence simulation via learned phase-to-space transform[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 14759-14768.
[10] 黄建明, 沈忙作. 基于噪声特性的大气湍流退化图像多帧盲反卷积复原[J]. 光学学报, 2008, 28(9): 1686-1690.
HUANG J M, SHEN M Z. Multi frame blind deconvolution restoration of atmospheric turbulence-degraded images based on noise characteristic[J]. Acta Optica Sinica, 2008, 28(9): 1686-1690.
[11] LAW N M. Lucky imaging: diffraction-limited astronomy from the ground in the visible[J]. The Observatory, 2007, 127: 71-71.
[12] HIRSCH M, SRA S, SCHOLKOPF B, et al. Efficient filter flow for space-variant multi frame blinddeconvolution[C]//Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010: 607-614.
[13] LAU C P, SOURI H, CHELLAPPA R. ATFaceGAN: single face image restoration and recognitionfrom atmospheric turbulence[C]//Proceedings of the 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition, 2020: 32-39.
[14] YASARLA R, PATEL V M. Learning to restore a single face image degraded by atmospheric turbulence using cnns[J]. arXiv:2007.08404, 2020.
[15] YASARLA R, PATEL V M. Learning to restore images degraded by atmospheric turbulence using uncertainty[C]//Proceedings of the 2021 IEEE International Conference on Image Processing, 2021: 1694-1698.
[16] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 6000-6010.
[17] LEE J, KIM J. Improving video captioning with non-local neural networks[C]//Proceedings of the 2018 IEEE International Conference on Consumer Electronics-Asia, 2018: 206-212.
[18] ZHANG H, GOODFELLOW I, METAXAS D, et al. Self-attention generative adversarial networks[C]//Proceedings of the International Conference on Machine Learning, 2019: 7354-7363.
[19] KENDALL A, GAL Y. What uncertainties do we need in Bayesian deep learning for computer vision?[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 5580-5590.
[20] WANG Y, BAI X. Wide & deep learning for spatial & intensity adaptive image restoration[J]. arXiv:2305.18708, 2023.
[21] MAO Z Y, JAISWAL A, WANG Z, et al. Single frame atmospheric turbulence mitigation: a benchmark study and a new physics-inspired transformer model[C]//Proceedings of the European Conference on Computer Vision, 2022: 430-446.
[22] ZAMIR S W, ARORA A, KHAN S, et al. Multi-stage progressive image restoration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 14821-14831.
[23] AL-NAJJAR Y, CHEN D. Comparison of image quality assessment: PSNR, HVS, SSIM, UIQI[J]. International Journal of Scientific and Engineering Research, 2012, 3(8): 1-5.
[24] SHEIKH H R, BOVIK A C, DE V G. An information fidelity criterion for image quality assessment using natural scene statistics[J]. IEEE Transactions on Image Processing, 2005, 14(12): 2117-2128.
[25] NIRANJAN D V, THOMAS D K, WILSON S G, et al. Image quality assessment based on a degradation model[J]. IEEE Transactions on Image Processing, 2000, 9(4): 636-650.
[26] RIM J, LEE H, WON J, et al. Real-world blur dataset for learning and benchmarking deblurring algorithms[C]//Proceedings of the European Conference on Computer Vision, 2020: 23-28.