Review of Deep Learning Methods Applied to Medical CT Super-Resolution

doi:10.3778/j.issn.1002-8331.2303-0224

Abstract

Abstract: Image super resolution (SR) is one of the important processing methods to improve image resolution in the field of computer vision, which has important research significance and application value in the field of medical image. High quality and high-resolution medical CT images are very important in the current clinical process. In recent years, the technology of medical CT image super-resolution reconstruction based on deep learning has made remarkable progress. This paper reviews the representative methods in this field and systematically reviews the development of medical CT image super-resolution reconstruction technology. Firstly, the basic theory of SR is introduced, and the commonly used evaluation indexes are given. Then, it focuses on the innovation and progress of super-resolution reconstruction of medical CT images based on deep learning, and makes a comprehensive comparative analysis of the main characteristics and performance of each method. Finally, the difficulties and challenges in the direction of medical CT image super-resolution reconstruction are discussed, and the future development trend is summarized and prospected, hoping to provide reference for related research.

Key words: super resolution, medical CT images, deep learning, computer vision, neural network

摘要： 图像超分辨率（SR）是计算机视觉领域提高图像分辨率的重要处理方法之一，在医学图像领域有重要的研究意义和应用价值。高质量和高分辨率的医学CT影像在当前的临床过程中非常重要。近年来，基于深度学习的医学CT影像超分辨率重建技术取得了显著的进展，对该领域内的代表性方法进行了梳理，系统回顾了医学CT影像超分辨率重建技术的发展。介绍了SR基本理论，给出常用的评价指标；重点阐述基于深度学习的医学CT影像超分辨率重建方向的创新与进展，对各个方法的主要特点和性能进行了综合比较分析。最后，讨论了医学CT影像超分辨率重建方向上存在的困难和挑战，并对未来的发展趋势进行了总结与展望，希望能为相关研究提供参考。

关键词: 超分辨率, 医学CT影像, 深度学习, 计算机视觉, 神经网络

TIAN Miaomiao, ZHI Lijia, ZHANG Shaomin, CHAO Daifu. Review of Deep Learning Methods Applied to Medical CT Super-Resolution[J]. Computer Engineering and Applications, 2024, 60(3): 44-60.

田苗苗, 支力佳, 张少敏, 晁代福. 医学CT影像超分辨率深度学习方法综述[J]. 计算机工程与应用, 2024, 60(3): 44-60.

References

[1] YEH J J, CHEN S C C, TENG W B, et al. Identifying the most infectious lesions in pulmonary tuberculosis by high-resolution multi-detector computed tomography[J]. European Radiology, 2010, 20: 2135-2145.
[2] AKIRA M, INOUE Y, ARAI T, et al. Pulmonary fibrosis on high-resolution CT of patients with pulmonary alveolar proteinosis[J]. AJR American Journal of Roentgenology, 2016, 207(3): 544-551.
[3] LEE H Y, LEE K S, JEONG Y J, et al. High-resolution CT findings in fibrotic idiopathic interstitial pneumonias with little honeycombing: serial changes and prognostic implications[J]. AJR American Journal of Roentgenology, 2012, 199(5): 982-989.
[4] 刘东权.超高分辨与常规螺旋 CT 扫描对肺结节良恶性鉴别的临床观察[J]. 浙江创伤外科, 2020(6):1208-1210.
LIU D Q. Clinical observation of diagnosis of benign and malignant pulmonary nodules by ultra high resolution and conventional spiral CT[J]. Zhejiang Journal of Traumatic Surgery, 2020(6):1208-1210.
[5] GERCHBERG R W. Super-resolution through error energy reduction[J]. Optica Acta: International Journal of Optics, 1974, 21(9): 709-720.
[6] HOU H, JIN Q, ZHANG G, et al. CT image quality enhancement via a dual-channel neural network with jointing denoising and super-resolution[J]. Neurocomputing, 2022, 492: 343-352.
[7] BAKER S, KANADE T. Limits on super-resolution and how to break them[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(9): 1167-1183.
[8] DONG C, LOY C C, HE K, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307.
[9] CHAUDHARI A S, FANG Z, KOGAN F, et al. Super‐resolution musculoskeletal MRI using deep learning[J]. Magnetic Resonance in Medicine, 2018, 80(5): 2139-2154.
[10] MAHAPATRA D, BOZORGTABAR B, GARNAVI R. Image super-resolution using progressive generative adversarial networks for medical image analysis[J]. Computerized Medical Imaging and Graphics, 2019, 71: 30-39.
[11] BASHIR S M A, WANG Y. Deep learning for the assisted diagnosis of movement disorders, including isolated dystonia[J]. Frontiers in Neurology, 2021, 12: 638266.
[12] UMEHARA K, OTA J, ISHIDA T. Application of super-resolution convolutional neural network for enhancing image resolution in chest CT[J]. Journal of Digital Imaging, 2018, 31: 441-450.
[13] KIM J, LEE J K, LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1646-1654.
[14] KEYS R. Cubic convolution interpolation for digital image processing[J]. IEEE Transactions on Acoustics, Speech, and Signal Processing, 1981, 29(6): 1153-1160.
[15] HONG S H, WANG L, TRUONG T K. An improved approach to the cubic-spline interpolation[C]//Proceedings of the 25th IEEE International Conference on Image Processing (ICIP), 2018: 1468-1472.
[16] DONG C, LOY C C, TANG X. Accelerating the super-resolution convolutional neural network[C]//Proceedings of the 14th European Conference on Computer Vision, Amsterdam, The Netherlands, October 11-14, 2016: 391-407.
[17] LAI W S, HUANG J B, AHUJA N, et al. Deep Laplacian pyramid networks for fast and accurate super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 624-632.
[18] WANG Y, PERAZZI F, MCWILLIAMS B, et al. A fully progressive approach to single-image super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 864-873.
[19] SONG Z, ZHAO X, HUI Y, et al. Progressive back-projection network for COVID-CT super-resolution[J]. Computer Methods and Programs in Biomedicine, 2021, 208: 106193.
[20] LI Z, YANG J, LIU Z, et al. Feedback network for image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 3867-3876.
[21] MAO X, LI Q, XIE H, et al. Least squares generative adversarial networks[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 2794-2802.
[22] SAJJADI M S M, SCHOLKOPF B, HIRSCH M. Enhancenet: single image super-resolution through automated texture synthesis[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 4491-4500.
[23] STREIJL R C, WINKLER S, HANDS D S. Mean opinion score (MOS) revisited: methods and applications, limitations and alternatives[J]. Multimedia Systems, 2016, 22(2): 213-227.
[24] LUGMAYR A, DANELLJAN M, TIMOFTE R. Ntire 2020 challenge on real-world image super-resolution: methods and results[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 494-495.
[25] ZHOU W, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[26] PAMBRUN J F, NOUMEIR R. Limitations of the SSIM quality metric in the context of diagnostic imaging[C]//Proceedings of the IEEE International Conference on Image Processing (ICIP), 2015: 2960-2963.
[27] SHEIKH H R, BOVIK A C, VECIANA G D. An information fidelity criterion for image quality assessment using natural scene statistics[J]. IEEE Transactions on Image Processing, 2005, 14(12): 2117-2128.
[28] SHEIKH H R, BOVIK A C. Image information and visual quality[J]. IEEE Transactions on Image Processing, 2006, 15(2): 430-444.
[29] GEORGESCU M-I, IONESCU R T, VERGA N. Convolutional neural networks with intermediate loss for 3D super-resolution of CT and MRI scans[J]. IEEE Access, 2020, 8: 49112-49124.
[30] AMARANAGESWARAO G, DEIVALAKSHMI S, KO S B. Wavelet based medical image super resolution using cross connected residual-in-dense grouped convolutional neural network[J]. Journal of Visual Communication and Image Representation, 2020, 70:102819.
[31] GU Y, ZENG Z, CHEN H, et al. MedSRGAN: medical images super?resolution using generative adversarial networks[J]. Multimedia Tools and Applications, 2020, 79(29): 21815-21840.
[32] KUDO A, KITAMURA Y, LI Y, et al. Virtual thin slice: 3D conditional GAN-based super-resolution for CT slice interval[C]//Proceedings of the International Workshop on Machine Learning for Medical Image Reconstruction, Shenzhen, China, October 17, 2019: 91-100.
[33] ZHANG J, GONG L R, YU K, et al. 3D reconstruction for super-resolution CT images in the Internet of health things using deep learning[J]. IEEE Access, 2020, 8: 121513-121525.
[34] JI X, CAO Y, TAI Y, et al. Real-world super-resolution via kernel estimation and noise injection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 466-467.
[35] YOU C, LI G, ZHANG Y, et al. CT super-resolution GAN constrained by the identical, residual, and cycle learning ensemble (GAN-CIRCLE)[J]. IEEE Transactions on Medical Imaging, 2019, 39(1): 188-203.
[36] JIANG X, LIU M, ZHAO F, et al. A novel super-resolution CT image reconstruction via semi?supervised generative adversarial network[J]. Neural Computing and Applications, 2020, 32(18): 14563-14578.
[37] JIANG X, XU Y, WEI P, et al. CT image super resolution based on improved SRGAN[C]//Proceedings of the 5th International Conference on Computer and Communication Systems (ICCCS), 2020: 363-367.
[38] ZHANG X, FENG C, WANG A, et al. CT super-resolution using multiple dense residual block based GAN[J]. Signal, Image and Video Processing, 2021, 15(4): 725-733.
[39] MUSUNURI Y R, KWON O S.Deep residual dense network for single image super-resolution[J].Electronics, 2021, 10(5):555.
[40] HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 7132-7141.
[41] ZHAO T, HU L, ZHANG Y, et al. Super-resolution network with information distillation and multi-scale attention for medical CT image[J]. Sensors (Basel), 2021, 21: 6870.
[42] ZHANG Y, LI K, LI K, et al. Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018: 286-301.
[43] HARIS M, SHAKHNAROVICH G, UKITA N. Deep back-projection networks for super-resolution[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 1664-1673.
[44] QIU D, CHENG Y, WANG X, et al. Multi-window back-projection residual networks for reconstructing COVID-19 CT super-resolution images[J]. Comput Methods Programs Biomed, 2021, 200: 105934.
[45] RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, October 5-9, 2015: 234-241.
[46] QIU D, CHENG Y, WANG X. Progressive U-Net residual network for computed tomography images super-resolution in the screening of COVID?19[J]. Journal of Radiation Research and Applied Sciences, 2021, 14(1): 369-379.
[47] BING X, ZHANG W, ZHENG L, et al. Medical image super resolution using improved generative adversarial networks[J]. IEEE Access, 2019, 7: 145030-145038.
[48] YU P, ZHANG H, KANG H, et al. RPLHR-CT dataset and transformer baseline for volumetric super-resolution from CT scans[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2022: 344-353.
[49] IANDOLA F N, HAN S, MOSKEWICZ M W, et al. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 0.5 MB model size[J]. arXiv:1602.07360, 2016.
[50] LI Y, IWAMOTO Y, LIN L, et al. VolumeNet: a lightweight parallel network for super-resolution of MR and CT volumetric data[J]. IEEE Transactions on Image Process, 2021, 30: 4840-4854.
[51] SHI W, CABALLERO J, HUSZáR F, et al. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 1874-1883.
[52] ZEILER M D, KRISHNAN D, TAYLOR G W, et al. Deconvolutional networks[C]//Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010: 2528-2535.
[53] HU X, MU H, ZHANG X, et al. Meta-SR: a magnification-arbitrary network for super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 1575-1584.
[54] ZHU J, TAN C, YANG J, et al. Arbitrary scale super-resolution for medical images[J]. International Journal of Neural Systems, 2021, 31(10): 2150037.
[55] DE FARIAS E C, DI NOIA C, HAN C, et al. Impact of GAN-based lesion-focused medical image super-resolution on the robustness of radiomic features[J]. Scientific Reports, 2021, 11(1): 21361.
[56] 梁聪. 面向图像分类的卷积神经网络损失函数研究[D]. 济南：山东大学, 2020.
LIANG C. Study on convolutional neural network loss function for image classification[D].Jinan: Shandong University, 2020.
[57] JOLICOEUR-MARTINEAU A. The relativistic discriminator: a key element missing from standard GAN[J]. arXiv:1807.00734, 2018.
[58] WANG X, YU K, WU S, et al. EsrGAN: enhanced super-resolution generative adversarial networks[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[59] YUAN L, XU X. Adaptive image edge detection algorithm based on canny operator[C]//Proceedings of the 4th International Conference on Advanced Information Technology and Sensor Application (AITS), 2015: 28-31.
[60] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Advances in Neural Information Processing Systems, 2014.
[61] KHAN S, NASEER M, HAYAT M, et al. Transformers in vision: a survey[J]. ACM Computing Surveys (CSUR), 2022, 54: 1-41.
[62] MATSOUKAS C, HASLUM J F, S?DERBERG M, et al. Is it time to replace cnns with transformers for medical images?[J]. arXiv:2108.09038, 2021.
[63] LIU Z, LIN Y, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 10012-10022.
[64] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognition at scale[J]. arXiv:2010.11929, 2020.
[65] SETIO A A A, TRAVERSO A, DE BEL T, et al. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge[J]. Medical Image Analysis, 2017, 42: 1-13.
[66] YAN K, WANG X, LU L, et al. Deep lesion graphs in the wild: relationship learning and organization of significant radiology image findings in a diverse large-scale lesion database[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 9261-9270.
[67] ARBELAEZ P, MAIRE M, FOWLKES C, et al. Contour detection and hierarchical image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 33(5): 898-916.
[68] YANG X, HE X, ZHAO J, et al. COVID-CT-dataset: a CT scan dataset about COVID-19[J]. arXiv:2003.13865, 2020.
[69] AGUSTSSON E, TIMOFTE R. Ntire 2017 challenge on single image super-resolution: dataset and study[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 126-135.
[70] REHMAN S, MUNIR M K, IQBAL R. Novel coronavirus 2019 (nCoV)[J].Pakistan Journal of Medical Research, 2020, 59(1):1-2.
[71] CHEN Z, WANG R F. Application value of CT in the diagnosis and differential diagnosis of novel coronavirus pneumonia[J]. CT Theory & Applications, 2020, 29(3).
[72] TAN W, LIU P, LI X, et al. Classification of COVID-19 pneumonia from chest CT images based on reconstructed super-resolution images and VGG neural network[J]. Health Information Science and Systems, 2021, 9(1): 10.
[73] NNEJI G U, DENG J, MONDAY H N, et al. COVID-19 identification from low-quality computed tomography using a modified enhanced super-resolution generative adversarial network plus and siamese capsule network[J]. Healthcare (Basel), 2022, 10(2):403.
[74] BACCARELLI E, SCARPINITI M, MOMENZADEH A. Twinned residual auto-encoder (TRAE)—a new DL architecture for denoising super-resolution and task?aware feature learning from COVID-19 CT images[J]. Expert Systems with Applications, 2023, 225: 120104.
[75] ZHOU C, XIONG A. Fast image super-resolution using particle swarm optimization-based convolutional neural networks[J]. Sensors (Basel), 2023, 23(4).
[76] BRüLLMANN D, SCHULZE R K W. Spatial resolution in CBCT machines for dental/maxillofacial applications—what do we know today?[J]. DentoMaxillo Facial Radiology, 2015, 44(1): 20140204.
[77] HATVANI J, HORVáTH A, MICHETTI J, et al. Deep learning-based super-resolution applied to dental computed tomography[J]. IEEE Transactions on Radiation and Plasma Medical Sciences, 2018, 3(2): 120-128.
[78] HATVANI J, BASARAB A, TOURNERET J Y, et al. A tensor factorization method for 3-D Super resolution with application to dental CT[J]. IEEE Transactions on Medical Imaging, 2019, 38(6): 1524-1531.
[79] MOHAMMAD-RAHIMI H, VINAYAHALINGAM S, MAHMOUDINIA E, et al. Super-resolution of dental panoramic radiographs using deep learning: a pilot study[J]. Diagnostics (Basel), 2023, 13(5).
[80] WANG X, GUO C, ZHA Y, et al. Diagnosis of nonperitonealized colorectal cancer with computerized tomography image features under deep learning[J]. Contrast Media & Molecular Imaging, 2022, 2022: 1886406.
[81] LIU D, WANG S, WANG J. The effect of CT high-resolution imaging diagnosis based on deep residual network on the pathology of bladder cancer classification and staging[J]. Computer Methods and Programs in Biomedicine, 2022, 215: 106635.
[82] XU J, LIU W, QIN Y, et al. Image super-resolution reconstruction method for lung cancer CT-scanned images based on neural network[J]. BioMed Research International, 2022, 2022: 3543531.
[83] ZHU D, SUN D, WANG D. Dual attention mechanism network for lung cancer images super-resolution[J]. Computer Methods and Programs in Biomedicine, 2022, 226: 107101.
[84] REN H, EL-KHAMY M, LEE J. CT-SRCNN: cascade trained and trimmed deep convolutional neural networks for image super resolution[C]//Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV)， 2018: 1423-1431.
[85] PENG C, LIN W A, LIAO H, et al. SAINT: spatially aware interpolation network for medical slice synthesis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020: 7750-7759.
[86] ZHAO M, LIU X, LIU H, et al. Super-resolution of cardiac magnetic resonance images using Laplacian pyramid based on generative adversarial networks[J]. Computerized Medical Imaging and Graphics, 2020, 80: 101698.
[87] CHI J, ZHANG Y, YU X, et al. Computed tomography (CT) image quality enhancement via a uniform framework integrating noise estimation and super-resolution networks[J]. Sensors (Basel), 2019, 19(15):3348.