Review of Deep Learning Methods for MRI Reconstruction

doi:10.3778/j.issn.1002-8331.2302-0057

Abstract

Abstract: Magnetic resonance imaging（MRI） is a commonly used imaging technique in the clinic, but the excessive imaging time limits its further development. Image reconstruction from undersampled k-space data has been an important part of accelerating MRI imaging. In recent years, deep learning has shown great potential in MRI reconstruction, and its reconstruction results and efficiency are better than traditional compressed sensing methods. To sort out and summarize the current deep learning-based MRI reconstruction methods, it firstly introduces the definition of MRI reconstruction problem, secondly analyzes the application of deep learning in data-driven end-to-end reconstruction and model-driven unrolled optimization reconstruction, then provides evaluation metrics and common datasets for reconstruction, and finally discusses the challenges faced by current MRI reconstruction and future research directions.

Key words: magnetic resonance imaging（MRI）, deep learning, image reconstruction, neural network

摘要： 磁共振成像（MRI）是临床中一种常用的成像技术，但过长的成像时间限制了其进一步的发展。从欠采样的k空间数据中进行图像重建是加速MRI成像的重要一环。近年来，深度学习在MRI重建方面显示出巨大的潜力，其重建结果和效率都优于传统的压缩感知方法。为梳理与总结当前基于深度学习的MRI重建方法，介绍了MRI重建问题的定义，分析了深度学习在数据驱动的端到端重建和模型驱动的展开优化重建中的应用，提供重建的评价指标和常用数据集，讨论了当前MRI重建所面临的挑战与未来研究方向。

关键词: 磁共振成像（MRI）, 深度学习, 图像重建, 神经网络

DENG Gewen, WEI Guohui, MA Zhiqing. Review of Deep Learning Methods for MRI Reconstruction[J]. Computer Engineering and Applications, 2023, 59(20): 67-76.

邓戈文, 魏国辉, 马志庆. 基于深度学习的MRI重建方法综述[J]. 计算机工程与应用, 2023, 59(20): 67-76.

References

[1] PRUESSMANN K P，WEIGER M，SCHEIDEGGER M B，et al.SENSE：sensitivity encoding for fast MRI[J].Magnetic Resonance in Medicine，1999，42（5）：952-962.
[2] SODICKSON D K，MANNING W J.Simultaneous acquisition of spatial harmonics（SMASH）：fast imaging with radio frequency coil arrays[J].Magnetic Resonance in Medicine，1997，38（4）：591-603.
[3] GRISWOLD M A，JAKOB P M，HEIDEMANN R M，et al.Generalized autocalibrating partially parallel acquisitions（GRAPPA）[J].Magnetic Resonance in Medicine，2002，47（6）：1202-1210.
[4] LUSTIG M，PAULY J M.SPIRiT：iterative self-consistent parallel imaging reconstruction from arbitrary k-space[J].Magnetic Resonance in Medicine，2010，64（2）：457-471.
[5] BLOCK K T，UECKER M，FRAHM J.Undersampled radial MRI with multiple coils.Iterative image reconstruction using a total variation constraint[J].Magnetic Resonance in Medicine，2007，57（6）：1086-1098.
[6] LUSTIG M，DONOHO D，PAULY J M.Sparse MRI：the application of compressed sensing for rapid MR imaging[J].Magnetic Resonance in Medicine，2007，58（6）：1182-1195.
[7] LIN D J，JOHNSON P M，KNOLL F，et al.Artificial intelligence for MR image reconstruction：an overview for clinicians[J].Journal of Magnetic Resonance Imaging，2021，53（4）：1015-1028.
[8] BEN YEDDER H，CARDOEN B，HAMARNEH G.Deep learning for biomedical image reconstruction：a survey[J].Artificial Intelligence Review，2021，54（1）：215-251.
[9] LEE J G，JUN S，CHO Y W，et al.Deep learning in medical imaging：general overview[J].Korean Journal of Radiology，2017，18（4）：570-584.
[10] RUECKERT D，SCHNABEL J.Model-based and data-driven strategies in medical image computing[J].Proceedings of the IEEE，2020，108（1）：110-124.
[11] GHODRATI V，SHAO J，BYDDER M，et al.MR image reconstruction using deep learning：evaluation of network structure and loss functions[J].Quantitative Imaging in Medicine and Surgery，2019，9（9）：1516-1527.
[12] LIANG D，CHENG J，KE Z，et al.Deep magnetic resonance image reconstruction：inverse problems meet neural networks[J].IEEE Signal Processing Magazine，2020，37（1）：141-151.
[13] WANG S，SU Z，YING L，et al.Accelerating magnetic resonance imaging via deep learning[C]//2016 IEEE 13th International Symposium on Biomedical Imaging（ISBI），2016：514-517.
[14] MONTALT-TORDERA J，MUTHURANGU V，HAUPTMANN A，et al.Machine learning in magnetic resonance imaging：image reconstruction[J].Physica Medica，2021，83：79-87.
[15] PAL A，RATHI Y.A review and experimental evaluation of deep learning methods for MRI reconstruction[J].arXiv：2109.08618，2021.
[16] WANG S，XIAO T，LIU Q，et al.Deep learning for fast MR imaging：a review for learning reconstruction from incomplete k-space data[J].Biomedical Signal Processing and Control，2021，68：102579.
[17] ZHU B，LIU J Z，CAULEY S F，et al.Image reconstruction by domain-transform manifold learning[J].Nature，2018，555（7697）：487.
[18] COLE E K，CHENG J Y，PAULY J M，et al.Analysis of deep complex-valued convolutional neural networks for MRI reconstruction[J].arXiv：2004.01738，2020.
[19] GAN W，ELDENIZ C，LIU J，et al.Image reconstruction for MRI using deep CNN priors trained without groundtruth[J].arXiv：2204.04771，2022.
[20] HAUPTMANN A，ARRIDGE S，LUCKA F，et al.Real-time cardiovascular MR with spatio-temporal artifact suppression using deep learning-proof of concept in congenital heart disease[J].Magnetic Resonance in Medicine，2019，81（2）：1143-1156.
[21] HAN Y，SUNWOO L，YE J C.k-Space deep learning for accelerated MRI[J].IEEE Transactions on Medical Imaging，2020，39（2）：377-386.
[22] DHENGRE N，SINHA S.Multiscale U-net-based accele-
rated magnetic resonance imaging reconstruction[J].Signal，Image and Video Processing，2022，16（4）：881-888.
[23] CAI C，WANG C，ZENG Y，et al.Single-shot T2 mapping using overlapping-echo detachment planar imaging and a deep convolutional neural network[J].Magnetic Resonance in Medicine，2018，80（5）：2202-2214.
[24] BAO L，YE F，CAI C，et al.Undersampled MR image reconstruction using an enhanced recursive residual network[J].Journal of Magnetic Resonance，2019，305：232-246.
[25] DU J，HE Z，WANG L，et al.Super-resolution reconstruction of single anisotropic 3D MR images using residual convolutional neural network[J].Neurocomputing，2020，392：209-220.
[26] PAWAR K，CHEN Z，SHAH N J，et al.Suppressing motion artefacts in MRI using an Inception-ResNet network with motion simulation augmentation[J].NMR in Biomedicine，2022，35（4）：e4225.
[27] ZHANG T，JACKSON L H，UUS A，et al.Self-supervised recurrent neural network for 4D abdominal and in-utero MR imaging[C]//International Workshop on Machine Learning for Medical Image Reconstruction，2019：16-24.
[28] QIN C，SCHLEMPER J，CABALLERO J，et al.Convolutional recurrent neural networks for dynamic MR image reconstruction[J].IEEE Transactions on Medical Imaging，2019，38（1）：280-290.
[29] OH C，CHUNG J Y，HAN Y.An end-to-end recurrent neural network for radial MR image reconstruction[J].Sensors，2022，22（19）：7277.
[30] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[J].arXiv：1706.03762，2017.
[31] DOSOVITSKIY A，BEYER L，KOLESNIKOV A，et al.An image is worth 16x16 words：transformers for image recognition at scale[J].arXiv：2010.11929，2020.
[32] FENG C M，YAN Y，CHEN G，et al.Accelerated multi-modal MR imaging with transformers[J].arXiv：2106.14248，2021.
[33] FENG C M，YAN Y，FU H，et al.Task transformer network for joint MRI reconstruction and super-resolution[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention，2021：307-317.
[34] MAHAPATRA D，GE Z.MR image super resolution by combining feature disentanglement CNNs and vision transformers[C]//International Conference on Medical Imaging with Deep Learning，2022.
[35] PARK T，ZHU J Y，WANG O，et al.Swapping autoencoder for deep image manipulation[C]//Advances in Neural Information Processing Systems，2020：7198-7211.
[36] KORKMAZ Y，DAR S U，YURT M，et al.Unsupervised MRI reconstruction via zero-shot learned adversarial transformers[J].IEEE Transactions on Medical Imaging，2022，41（7）：1747-1763.
[37] KORKMAZ Y，YURT M，DAR S U H，et al.Deep MRI reconstruction with generative vision transformers[C]//International Workshop on Machine Learning for Medical Image Reconstruction，2021：54-64.
[38] LIN K，HECKEL R.Vision transformers enable fast and robust accelerated MRI[C]//International Conference on Medical Imaging with Deep Learning，2022.
[39] LIU Z，LIN Y，CAO Y，et al.Swin transformer：hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：10012-10022.
[40] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial networks[J].Communications of the ACM，2020，63（11）：139-144.
[41] SHENDE P，PAWAR M，KAKDE S.A brief review on：MRI images reconstruction using GAN[C]//2019 International Conference on Communication and Signal Processing（ICCSP），2019：139-142.
[42] YANG G，YU S，DONG H，et al.DAGAN：deep de-aliasing generative adversarial networks for fast compressed sensing MRI reconstruction[J].IEEE Transactions on Medical Imaging，2018，37（6）：1310-1321.
[43] MARDANI M，GONG E，CHENG J Y，et al.Deep generative adversarial neural networks for compressive sensing MRI[J].IEEE Transactions on Medical Imaging，2019，38（1）：167-179.
[44] QUAN T M，NGUYEN-DUC T，JEONG W K.Compressed sensing MRI reconstruction using a generative adversarial network with a cyclic loss[J].IEEE Transactions on Medical Imaging，2018，37（6）：1488-1497.
[45] JIANG M，YUAN Z，YANG X，et al.Accelerating CS-MRI reconstruction with fine-tuning Wasserstein generative adversarial network[J].IEEE Access，2019，7：152347-152357.
[46] SHAUL R，DAVID I，SHITRIT O，et al.Subsampled brain MRI reconstruction by generative adversarial neural networks[J].Medical Image Analysis，2020，65：101747.
[47] LV J，WANG C，YANG G.PIC-GAN：a parallel imaging coupled generative adversarial network for accelerated multi-channel MRI reconstruction[J].Diagnostics，2021，11（1）：61.
[48] HUANG J，DING W，LV J，et al.Edge-enhanced dual discriminator generative adversarial network for fast MRI with parallel imaging using multi-view information[J].Applied Intelligence，2022：1-18.
[49] MAKHZANI A，SHLENS J，JAITLY N，et al.Adversarial autoencoders[J].arXiv：1511.05644，2015.
[50] TEZCAN K C，BAUMGARTNER C F，LUECHINGER R，et al.MR image reconstruction using deep density priors[J].IEEE Transactions on Medical Imaging，2019，38（7）：1633-1642.
[51] ZOU Q，AHMED A H，NAGPAL P，et al.Dynamic imaging using a deep generative SToRM（Gen-SToRM） model[J].IEEE Transactions on Medical Imaging，2021，40（11）：3102-3112.
[52] HIGGINS I，MATTHEY L，PAL A，et al.beta-VAE：learning basic visual concepts with a constrained variational framework[C]//International Conference on Learning Representations，2016.
[53] ZOU Q，AHMED A H，NAGPAL P，et al.Deep generative storm model for dynamic imaging[C]//2021 IEEE 18th International Symposium on Biomedical Imaging（ISBI），2021：114-117.
[54] CHUNG H，YE J C.Score-based diffusion models for accelerated MRI[J].Medical Image Analysis，2022：102479.
[55] SONG Y，SHEN L，XING L，et al.Solving inverse problems in medical imaging with score-based generative models[J].arXiv：2111.08005，2021.
[56] CHUNG H，LEE E S，YE J C.MR image denoising and super-resolution using regularized reverse diffusion[J].IEEE Transactions on Medical Imaging，2023，42（4）：922-934.
[57] CHEN E Z，WANG P，CHEN X，et al.Pyramid convolutional RNN for MRI image reconstruction[J].IEEE Transactions on Medical Imaging，2022，41（8）：2033-2047.
[58] HUANG J，XING X，GAO Z，et al.Swin deformable attention U-Net transformer（SDAUT） for explainable fast MRI[C]//Medical Image Computing and Computer Assisted Intervention-MICCAI 2022：25th International Conference，Singapore，September 18-22，2022：538-548.
[59] ZHAO X，YANG T，LI B，et al.SwinGAN：a dual-domain swin transformer-based generative adversarial network for MRI reconstruction[J].Computers in Biology and Medicine，2023，153：106513.
[60] HAMMERNIK K，KLATZER T，KOBLER E，et al.Learning a variational network for reconstruction of accelerated MRI data[J].Magnetic Resonance in Medicine，2018，79（6）：3055-3071.
[61] SCHLEMPER J，CABALLERO J，HAJNAL J V，et al.A deep cascade of convolutional neural networks for dynamic MR image reconstruction[J].IEEE Transactions on Medical Imaging，2018，37（2）：491-503.
[62] HOSSEINI S A H，YAMAN B，MOELLER S，et al.Dense recurrent neural networks for accelerated MRI：history-cognizant unrolling of optimization algorithms[J].IEEE Journal of Selected Topics in Signal Processing，2020，14（6）：1280-1291.
[63] MARDANI M，MONAJEMI H，PAPYAN V，et al.Recurrent generative adversarial networks for proximal learning and automated compressive image recovery[J].arXiv：1711.10046，2017.
[64] ZHANG J，GHANEM B.ISTA-Net：interpretable optimization-inspired deep network for image compressive sensing[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：1828-1837.
[65] AGGARWAL H K，MANI M P，JACOB M.MoDL：model-based deep learning architecture for inverse problems[J].IEEE Transactions on Medical Imaging，2019，38（2）：394-405.
[66] DUAN J，SCHLEMPER J，QIN C，et al.VS-Net：variable splitting network for accelerated parallel MRI reconstruction[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention，2019：713-722.
[67] YANG Y，SUN J，LI H，et al.ADMM-Net：a deep learning approach for compressive sensing MRI[J].arXiv：1705.06869，2017.
[68] YANG Y，SUN J，LI H，et al.ADMM-CSNet：a deep learning approach for image compressive sensing[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2020，42（3）：521-538.
[69] ZHANG X，LIAN Q，YANG Y，et al.A deep unrolling network inspired by total variation for compressed sensing MRI[J].Digital Signal Processing，2020，107：102856.
[70] BISWAS S，AGGARWAL H K，JACOB M.Dynamic MRI using model-based deep learning and SToRM priors：MoDL-SToRM[J].Magnetic Resonance in Medicine，2019，82（1）：485-494.
[71] CHENG J，WANG H，YING L，et al.Model learning：primal dual networks for fast MR imaging[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention，2019：21-29.
[72] CHENG J，WANG H，ZHU Y，et al.Model-based deep medical imaging：the roadmap of generalizing iterative reconstruction model using deep learning[J].arXiv：1906.
08143，2019.
[73] VAN ESSEN D C，UGURBIL K，AUERBACH E，et al.The human connectome project：a data acquisition perspective[J].Neuroimage，2012，62（4）：2222-2231.
[74] ZBONTAR J，KNOLL F，SRIRAM A，et al.fastMRI：an open dataset and benchmarks for accelerated MRI[J].arXiv：1811.08839，2018.
[75] SOUZA R，LUCENA O，GARRAFA J，et al.An open，multi-vendor，multi-field-strength brain MR dataset and analysis of publicly available skull stripping methods agreement[J].Neuroimage，2018，170：482-494.
[76] MENZE B H，JAKAB A，BAUER S，et al.The multimodal brain tumor image segmentation benchmark（BRATS）[J].IEEE Transactions on Medical Imaging，2015，34（10）：1993-2024.
[77] BERNARD O，LALANDE A，ZOTTI C，et al.Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis：is the problem solved?[J].IEEE Transactions on Medical Imaging，2018，37（11）：2514-2525.
[78] EO T，JUN Y，KIM T，et al.KIKI-net：cross-domain convolutional neural networks for reconstructing undersampled magnetic resonance images[J].Magnetic Resonance in Medicine，2018，80（5）：2188-2201.
[79] LV J，CHEN K，YANG M，et al.Reconstruction of undersampled radial free-breathing 3D abdominal MRI using stacked convolutional auto-encoders[J].Medical Physics，2018，45（5）：2023-2032.
[80] LI X L，THICKSTUN J，GULRAJANI I，et al.Diffusion-LM improves controllable text generation[J].arXiv：2205.14217，2022.
[81] BREMPONG E A，KORNBLITH S，CHEN T，et al.Denoising pretraining for semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：4175-4186.
[82] ALCARAZ J M L，STRODTHOFF N.Diffusion-based time series imputation and forecasting with structured state space models[J].arXiv：2208.09399，2022.
[83] AVRAHAMI O，LISCHINSKI D，FRIED O.Blended diffusion for text-driven editing of natural images[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022：18208-18218.