Research Progress of Deep Learning in Spine Image Segmentation

doi:10.3778/j.issn.1002-8331.2211-0219

Abstract

Abstract: Imaging is an important basis for clinical diagnosis and treatment of spinal diseases. The spine has a complex structure, surrounded by muscles, blood vessels and nerve tissues, and the shape of the spine is easily changed due to pathological changes, so accurate segmentation of spine images can assist doctors to accurately locate lesions, efficiently evaluate lesions and guide surgical treatment. In recent years, deep learning has been widely used in the field of spine segmentation with its powerful feature learning capability. To study the current research status and development of deep learning in spine image segmentation tasks, this paper compares the classical frameworks of deep learning, organizes the commonly used datasets and evaluation metrics in spine image segmentation, summarizes the latest progress of different network models of CNN, FCN, U-Net and GAN applied in spine segmentation, analyzes the characteristics of the models on spine segmentation, and discusses in detail the current deep learning in spine segmentation, discusses in detail the current problems and challenges encountered in deep learning in spine segmentation, and presents an outlook on feasible future development directions.

Key words: deep learning, neural network, spine segmentation, image segmentation

摘要： 影像学检查是临床诊断和治疗脊椎疾病的重要依据，脊椎的结构复杂，周围布满肌肉、血管和神经组织，且由于病理变化容易导致脊椎形状发生改变，对脊椎图像的精确分割能够辅助医生精准定位病灶，高效评估病变，引导手术治疗等。近年来，深度学习凭借其强大的特征学习能力在脊椎分割领域得到广泛的应用。为研究深度学习在脊椎图像分割任务中的研究现状和发展，对深度学习的经典框架进行比较，将脊椎图像分割中常用的数据集与评价指标归纳介绍，总结CNN、FCN、U-Net和GAN不同网络模型在脊椎分割中应用的最新进展，对模型在脊椎分割上的特点进行分析，详细讨论了当前深度学习在脊椎分割中遇到的问题和挑战，并对今后的发展方向做出展望。

关键词: 深度学习, 神经网络, 脊椎分割, 图像分割

DIAO Yi, ZHANG Kuixing, JIANG Mei, XU Yunfeng, WEI Benzheng. Research Progress of Deep Learning in Spine Image Segmentation[J]. Computer Engineering and Applications, 2023, 59(18): 49-58.

刁毅, 张魁星, 江梅, 徐云峰, 魏本征. 深度学习在脊椎图像分割中的研究进展[J]. 计算机工程与应用, 2023, 59(18): 49-58.

References

[1] XIANG Q，ZHAO Y，LIN J，et al.Epigenetic modifications in spinal ligament aging[J].Ageing Research Reviews，2022，77：101598.
[2] XIAO L X，LIU C S，ZHONG S Z，et al.Effect of a traction exercise neck brace on cervical spondylopathy radiculopathy：a clinical study and finite element analysis[J].Evidence-Based Complementary and Alternative Medicine，2021：8825150.
[3] WANG Y，CHE M，XIN J，et al.The role of IL-1β and TNF-α in intervertebral disc degeneration[J].Biomedicine & Pharmacotherapy，2020，131：110660.
[4] LAVERDIèRE C，GEORGIOPOULOS M，AMES C P，et al.Adult spinal deformity surgery and frailty：a systematic review[J].Global Spine Journal，2022，12（4）：689-699.
[5] 郭春麟，张勇，刘祎，等.脊柱X射线图像分割方法[J].国外电子测量技术，2022，41（7）：23-28.
GUO C L，ZHANG Y，LIU Y，et al.Spinal X-ray image segmentation method[J].Foreign Electronic Measurement Technology，2022，41（7）：23-28.
[6] SINHA A，SPALKIT S，PRABHAKAR A，et al.Radiology in TB spine（X-rays，Ultrasound，CT，MRI）[M]//Tuberculosis of the spine.Singapore：Springer，2022：91-112.
[7] LEE H M，KIM Y J，CHO J B，et al.Computer-aided diagnosis for determining sagittal spinal curvatures using deep learning and radiography[J].Journal of Digital Imaging，2022，35：846-859.
[8] SONG Y，REN S，LU Y，et al.Deep learning-based automatic segmentation of images in cardiac radiography：a promising challenge[J].Computer Methods and Programs in Biomedicine，2022，220：106821.
[9] LIU X，SONG L，LIU S，et al.A review of deep-learning-based medical image segmentation methods[J].Sustainability，2021，13（3）：1224.
[10] ATTALLAH O.Deep learning-based CAD system for COVID-19 diagnosis via spectral-temporal images[C]//Proceedings of the 12th International Conference on Information Communication and Management，2022：25-33.
[11] QU B，CAO J，QIAN C，et al.Current development and prospects of deep learning in spine image analysis：a literature review[J].Quantiative Imaging in Medicine Surgery，2022，12（6）：3454-3479.
[12] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409.
1556，2014.
[13] SZEGEDY C，LIU W，JIA Y，et al.Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：1-9.
[14] SRIVASTAVA R K，GREFF K，SCHMIDHUBER J.Training very deep networks[C]//Advances in Neural Information Processing Systems，2015.
[15] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[16] HUANG G，LIU Z，VAN DER MAATEN L，et al.Densely connected convolutional networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：4700-4708.
[17] XIE S，GIRSHICK R，DOLLáR P，et al.Aggregated residual transformations for deep neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：1492-1500.
[18] HU J，SHEN L，SUN G.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：7132-7141.
[19] REDMON J，DIVVALA S，GIRSHICK R，et al.You only look once：unified，real-time object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：779-788.
[20] RONNEBERGER O，FISCHER P，BROX T.U-net：convolutional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham：Springer，2015：234-241.
[21] BADRINARAYANAN V，KENDALL A，CIPOLLA R.Segnet：a deep convolutional encoder-decoder architecture for image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，39（12）：2481-2495.
[22] HE K，GKIOXARI G，DOLLáR P，et al.Mask R-CNN[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2961-2969.
[23] CHEN L C，ZHU Y，PAPANDREOU G，et al.Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：801-818.
[24] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial networks[J].Communications of the ACM，2020，63（11）：139-144.
[25] DARGAN S，KUMAR M，AYYAGARI M R，et al.A survey of deep learning and its applications：a new paradigm to machine learning[J].Archives of Computational Methods in Engineering，2020，27（4）：1071-1092.
[26] RAK M，STEFFEN J，MEYER A，et al.Combining convolutional neural networks and star convex cuts for fast whole spine vertebra segmentation in MRI[J].Computer Methods and Programs in Biomedicine，2019，177：47-56.
[27] QADRI S F，SHEN L，AHMAD M，et al.OP-convNet：a patch classification-based framework for CT vertebrae segmentation[J].IEEE Access，2021，9：158227-158240.
[28] QADRI S F，SHEN L，AHMAD M，et al.SVseg：stacked sparse autoencoder-based patch classification modeling for vertebrae segmentation[J].Mathematics，2022，10（5）：796.
[29] KERVADEC H，DOLZ J，TANG M，et al.Constrained-CNN losses for weakly supervised segmentation[J].Medical Image Analysis，2019，54：88-99.
[30] TAM C M，ZHANG D，CHEN B，et al.Holistic multitask regression network for multiapplication shape regression segmentation[J].Medical Image Analysis，2020，65：101783.
[31] HONG Y，WEI B，HAN Z，et al.MMCL-Net：spinal disease diagnosis in global mode using progressive multi-task joint learning[J].Neurocomputing，2020，399：307-316.
[32] HUANG M，ZHOU S，CHEN X，et al.Semi-supervised hybrid spine network for segmentation of spine MR images[J].arXiv：2203.12151，2022.
[33] LAVDAS I，GLOCKER B，KAMNITSAS K，et al.Fully automatic，multiorgan segmentation in normal whole body magnetic resonance imaging（MRI），using classification forests（CFs），convolutional neural networks（CNNs），and a multi‐atlas（MA） approach[J].Medical Physics，2017，44（10）：5210-5220.
[34] IRIONDO C，PEDOIA V，MAJUMDAR S.Lumbar intervertebral disc characterization through quantitative MRI analysis：an automatic voxel‐based relaxometry approach[J].Magnetic Resonance in Medicine，2020，84（3）：1376-1390.
[35] SURI A，JONES B C，NG G，et al.A deep learning system for automated，multi-modality 2D segmentation of vertebral bodies and intervertebral discs[J].Bone，2021，149：115972.
[36] LONG J，SHELHAMER E，DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：3431-3440.
[37] CHEN H，DOU Q，WANG X，et al.3D fully convolutional networks for intervertebral disc localization and segmentation[C]//International Conference on Medical Imaging and Augmented Reality.Cham：Springer，2016：375-382.
[38] LI X，DOU Q，CHEN H，et al.3D multi-scale FCN with random modality voxel dropout learning for intervertebral disc localization and segmentation from multi-modality MR images[J].Medical Image Analysis，2018，45：41-54.
[39] ZENG G，ZHENG G.DSMS-FCN：a deeply supervised multi-scale fully convolutional network for automatic segmentation of intervertebral disc in 3D MR images[C]//International Workshop on Computational Methods and Clinical Applications in Musculoskeletal Imaging.Cham：Springer，2017：148-159.
[40] DAS P，PAL C，ACHARYYA A，et al.Deep neural network for automated simultaneous intervertebral disc（IVDs） identification and segmentation of multi-modal MR images[J].Computer Methods and Programs in Biomedicine，2021，205：106074.
[41] SEKUBOYINA A，KUKA?KA J，KIRSCHKE J S，et al.Attention-driven deep learning for pathological spine segmentation[C]//International Workshop on Computational Methods and Clinical Applications in Musculoskeletal Imaging.Cham：Springer，2017：108-119.
[42] XIA L，XIAO L，QUAN G，et al.3D cascaded convolutional networks for multi-vertebrae segmentation[J].Current Medical Imaging，2020，16（3）：231-240.
[43] JANSSENS R，ZENG G，ZHENG G.Fully automatic segmentation of lumbar vertebrae from CT images using cascaded 3D fully convolutional networks[C]//2018 IEEE 15th International Symposium on Biomedical Imaging（ISBI 2018），2018：893-897.
[44] AL ARIF S M M R，KNAPP K，SLABAUGH G.Fully automatic cervical vertebrae segmentation framework for X-ray images[J].Computer Methods and Programs in Biomedicine，2018，157：95-111.
[45] LESSMANN N，VAN GINNEKEN B，DE JONG P A，et al.Iterative fully convolutional neural networks for automatic vertebra segmentation and identification[J].Medical Image Analysis，2019，53：142-155.
[46] CHUANG C H，LIN C Y，TSAI Y Y，et al.Efficient triple output network for vertebral segmentation and identification[J].IEEE Access，2019，7：117978-117985.
[47] REHMAN F，ALI SHAH S I，RIAZ M N，et al.A region-based deep level set formulation for vertebral bone segmentation of osteoporotic fractures[J].Journal of Digital Imaging，2020，33（1）：191-203.
[48] ZHANG D，CHEN B，LI S.Sequential conditional reinforcement learning for simultaneous vertebral body detection and segmentation with modeling the spine anatomy[J].Medical Image Analysis，2021，67：101861.
[49] CHENG P，YANG Y，YU H，et al.Automatic vertebrae localization and segmentation in CT with a two-stage Dense-U-Net[J].Scientific Reports，2021，11（1）：1-13.
[50] 普钟，张俊华，黄昆，等.融合多注意力机制的脊椎图像分割方法[J].计算机应用研究，2023，40（4）：1256-1262.
PU Z，ZHANG J H，HUANG K，et al.Spinal image segmentation method with multi-attention[J].Application Research of Computers，2023，40（4）：1256-1262.
[51] ZHANG X，YANG Y，SHEN Y W，et al.SeUneter：channel attentive U-Net for instance segmentation of the cervical spine MRI medical image[J].Frontiers in Physiology，2022，13：2564.
[52] PANG S，PANG C，ZHAO L，et al.SpineParseNet：spine parsing for volumetric MR image by a two-stage segmentation framework with semantic image representation[J].IEEE Transactions on Medical Imaging，2020，40（1）：262-273.
[53] PANG S，PANG C，SU Z，et al.DGMSNet：spine segmentation for MR image by a detection-guided mixed-supervised segmentation network[J].Medical Image Analysis，2022，75：102261.
[54] LI H，LUO H，HUAN W，et al.Automatic lumbar spinal MRI image segmentation with a multi-scale attention network[J].Neural Computing and Applications，2021，33（18）：11589-11602.
[55] ZHOU J，DAMASCENO P F，CHACHAD R，et al.Automatic vertebral body segmentation based on deep learning of Dixon images for bone marrow fat fraction quantification[J].Frontiers in Endocrinology，2020，11：612.
[56] FANG Y，LI W，CHEN X，et al.Opportunistic osteoporosis screening in multi-detector CT images using deep convolutional neural networks[J].European Radiology，2021，31（4）：1831-1842.
[57] NAZIR A，CHEEMA M N，SHENG B，et al.ECSU-Net：an embedded clustering sliced U-Net coupled with fusing strategy for efficient intervertebral disc segmentation and classification[J].IEEE Transactions on Image Processing，2021，31：880-893.
[58] 程敏，沈林鹏，罗作煌.多维度卷积的两阶段脊柱三维实例分割方法[J].传感器与微系统，2022，41（12）：134-138.
CHENG M，SHEN L P，LUO Z H.Two-stage spine 3D instance segmentation method based on multi-dimensional convolutional[J].Transducer and Microsystem Technologies，2022，41（12）：134-138.
[59] 师文博，杨环，西永明，等.基于自注意力的双通路全脊柱X光图像分割模型[J].中国医学物理学杂志，2022，39（11）：1385-1392.
SHI W B，YANG H，XI Y M，et al.Self-attention based dual pathway network for spine segmentation in X-ray image[J].Chinese Journal of Medical Physics，2022，39（11）：1385-1392.
[60] YANG Z，WANG Q，ZENG J，et al.RAU-Net：U-Net network based on residual multi-scale fusion and attention skip layer for overall spine segmentation[J].Machine Vision and Applications，2023，34（1）：1-17.
[61] LUC P，COUPRIE C，CHINTALA S，et al.Semantic segmentation using adversarial networks[J].arXiv：1611.08408，2016.
[62] HAN Z，WEI B，MERCADO A，et al.Spine-GAN：semantic segmentation of multiple spinal structures[J].Medical Image Analysis，2018，50：23-35.
[63] GONG H，LIU J，LI S，et al.Axial-SpineGAN：simultaneous segmentation and diagnosis of multiple spinal structures on axial magnetic resonance imaging images[J].Physics in Medicine & Biology，2021，66（11）：115014.
[64] HUANG Z，ZHAO R，LEUNG F H F，et al.Joint spine segmentation and noise removal from ultrasound volume projection images with selective feature sharing[J].IEEE Transactions on Medical Imaging，2022，41（7）：1610-1624.
[65] GOODFELLOW I，BENGIO Y，COURVILLE A.Deep learning[M].[S.l.]：MIT Press，2016.
[66] LEI Y，LIU Y，DONG X，et al.Automatic multi-organ segmentation in thorax CT images using U-Net-GAN[C]//Medical Imaging 2019：Computer-Aided Diagnosis（SPIE，2019），2019：262-267.
[67] DONG X，LEI Y，WANG T，et al.Automatic multiorgan segmentation in thorax CT images using U‐net‐GAN[J].Medical Physics，2019，46（5）：2157-2168.
[68] YANG C J，LIN C L，WANG C K，et al.Generative adversarial network（GAN） for automatic reconstruction of the 3D spine structure by using simulated bi-planar X-ray images[J].Diagnostics，2022，12（5）：1121.
[69] KER J，WANG L，RAO J，et al.Deep learning applications in medical image analysis[J].IEEE Access，2017，6：9375-9389.
[70] ZHUANG F，QI Z，DUAN K，et al.A comprehensive survey on transfer learning[J].Proceedings of the IEEE，2020，109（1）：43-76.
[71] CHAKRABORTY S，TOMSETT R，RAGHAVENDRA R，et al.Interpretability of deep learning models：a survey of results[C]//2017 IEEE Smartworld，Ubiquitous Intelligence & Computing，Advanced & Trusted Computed，Scalable Computing & Communications，Cloud & Big Data Computing，Internet of People and Smart City Innovation（Smartworld/SCALCOM/UIC/ATC/CBDcom/IOP/SCI），2017：1-6.
[72] FORD R A，PRICE W，NICHOLSON I I.Privacy and accountability in black-box medicine[J].Michigan Telecommications & Technology Law Review，2016，23（1）.