Systematic Review on Graph Deep Learning in Medical Image Segmentation

doi:10.3778/j.issn.1002-8331.2112-0225

Abstract

Abstract: High precision segmentation of organs or lesions in medical image is a vital challenge issue for intelligent analysis of medical image, it has important clinical application value for auxiliary diagnosis and treatment of diseases. Recently, in solving challenging problems such as medical image information representation and accurate modeling of non-Euclidean spatial physiological tissue structures, the graph deep learning based medical image segmentation technology has made important breakthroughs, and it has shown significant information feature extraction and characterization advantages. The merged technology also can obtain more accurate segmentation results, which has become an emerging research hotspot in this field. In order to better promote the research and development of the deep learning segmentation algorithm for medical image graphs, this paper makes a systematic summary of the technological progress and application status in this field. The paper introduces the definition of graphs and the basic structure of graph convolutional networks, and elaborates on spectral graph convolution and spatial graph convolution operations. Then, according to the three technical structure modes of GCN combined with residual module, attention mechanism module and learning module, the research progress in medical image segmentation has been encapsulated. The application and development of graph deep learning algorithms based medical image segmentation are summarized and prospected to provide references and guiding principles for the technical development of related researches.

Key words: graph deep learning, graph neural network, graph convolutional network, medical image segmentation

摘要： 精准分割医学图像中的器官或病灶，是医学图像智能分析领域的重要难题，其在临床上对于疾病的辅助诊疗有着重要应用价值。在解决医学图像信息表征及对非欧空间生理组织结构准确建模等挑战性问题方面，基于图深度学习的医学图像分割技术取得了重要突破，展现出显著的信息特征提取及表征优势，可获得更为精准的分割结果，已成为该领域新兴研究热点。为更好促进医学图像图深度学习分割算法的研究发展，对该领域的技术进展及应用现状做了系统的梳理总结。介绍了图的定义及图卷积网络的基本结构，详细阐述了谱图卷积和空域图卷积操作。根据GCN结合残差模块、注意力机制模块及学习模块三种技术结构模式，归纳并总结了其在医学图像分割中的研究进展。对图深度学习算法在医学图像分割领域的应用和发展做了概要总结和展望，为该领域的技术发展提供参考和新的研究思路。

关键词: 图深度学习, 图神经网络, 图卷积网络, 医学图像分割

WANG Guoli, SUN Yu, WEI Benzheng. Systematic Review on Graph Deep Learning in Medical Image Segmentation[J]. Computer Engineering and Applications, 2022, 58(12): 37-50.

王国力, 孙宇, 魏本征. 医学图像图深度学习分割算法综述[J]. 计算机工程与应用, 2022, 58(12): 37-50.

References

[1] SUTTON R T，PINCOCK D，BAUMGART D C，et al.An overview of clinical decision support systems：benefits，risks，and strategies for success[J].NPJ Digital Medicine，2020，3（1）：1-10.
[2] JHA D，ALI S，TOMAR N K，et al.Real-time polyp detection，localization and segmentation in colonoscopy using deep learning[J].IEEE Access，2021，9：40496-40510.
[3] ALAKUS T B，TURKOGLU I.Comparison of deep learning approaches to predict COVID-19 infection[J].Chaos，Solitons & Fractals，2020，140：110120.
[4] GAO J，GUO Y，SUN Y，et al.Application of deep learning for early screening of colorectal precancerous lesions under white light endoscopy[J].Computational and Mathematical Methods in Medicine，2020，2020：1670-1748.
[5] MORTAZI A，BAGCI U.Automatically designing CNN architectures for medical image segmentation[C]//International Workshop on Machine Learning in Medical Imaging，2018：98-106.
[6] LI Q，CAI W，WANG X，et al.Medical image classification with convolutional neural network[C]//2014 13th International Conference on Control Automation Robotics & Vision（ICARCV），2014：844-848.
[7] NOOTHOUT J M，DE VOS B D，WOLTERINK J M，et al.CNN-based landmark detection in cardiac CTA scans[J].arXiv：1804.04963，2018.
[8] KISILEV P，SASON E，BARKAN E，et al.Medical image description using multi-task-loss CNN[M]//Deep learning and data labeling for medical applications.[S.l.]：Springer，2016：121-129.
[9] SCARSELLI F，GORI M，TSOI A C，et al.The graph neural network model[J].IEEE Transactions on Neural Networks，2008，20（1）：61-80.
[10] SINGERMAN D.On the structure of non-Euclidean crystallographic groups[C]//Mathematical Proceedings of the Cambridge Philosophical Society，1974：233-240.
[11] ESTRACH J B，ZAREMBA W，SZLAM A，et al.Spectral networks and deep locally connected networks on graphs[C]//2nd International Conference on Learning Representations，2014.
[12] ZHANG Y，LI Y，KONG Y，et al.GSCFN：a graph self-construction and fusion network for semi-supervised brain tissue segmentation in MRI[J].Neurocomputing，2021，455：23-37.
[13] ZHANG Y，KONG Y Y，WU J S，et al.Brain tissue segmentation based on graph convolutional networks[C]//2019 IEEE International Conference on Image Processing（ICIP），2019：1470-1474.
[14] TAN Z，FENG J，ZHOU J.SGNet：structure-aware graph-based network for airway semantic segmentation[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention，2021：153-163.
[15] TIAN F，GAO Y，FANG Z，et al.Automatic coronary artery segmentation algorithm based on deep learning and digital image processing[J].Applied Intelligence，2021，51：8881-8895.
[16] LI J，FENG C，LIN X，et al.Utilizing GCN and meta-learning strategy in unsupervised domain adaptation for pancreatic cancer segmentation[J].IEEE Journal of Biomedical and Health Informatics，2022，26（1）.
[17] BALISTRERI C R，CANDORE G，LIO D，et al.Prostate cancer：from the pathophysiologic implications of some genetic risk factors to translation in personalized cancer treatments[J].Cancer Gene Therapy，2014，21（1）：2-11.
[18] DEFFERRARD M，BRESSON X，VANDERGHEYNST P.Convolutional neural networks on graphs with fast localized spectral filtering[J].Advances in Neural Information Processing Systems，2016，29：3844-3852.
[19] KIPF T N，WELLING M.Semi-supervised classification with graph convolutional networks[J].arXiv：1609.02907，2016.
[20] NIEPERT M，AHMED M，KUTZKOV K.Learning convolutional neural networks for graphs[C]//International Conference on Machine Learning，2016：2014-2023.
[21] HAMILTON W L，YING R，LESKOVEC J.Inductive representation learning on large graphs[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems，2017：1025-1035.
[22] BRUNA J，ZAREMBA W，SZLAM A，et al.Spectral networks and locally connected networks on graphs[J].arXiv：1312.6203.2013.
[23] LI R，WANG S，ZHU F，et al.Adaptive graph convolutional neural networks[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2018.
[24] LEVIE R，MONTI F，BRESSON X，et al.Cayleynets：graph convolutional neural networks with complex rational spectral filters[J].IEEE Transactions on Signal Processing，2018，67（1）：97-109.
[25] CUI J，ZHUANG H，LIU T，et al.Semi-supervised gated spectral convolution on a directed signed network[J].IEEE Access，2020，8：49705-49716.
[26] FU S，LIU W，ZHANG K，et al.Semi-supervised classification by graph p-Laplacian convolutional networks[J].Information Sciences，2021，560：92-106.
[27] BIANCHI F M，GRATTAROLA D，LIVI L，et al.Graph neural networks with convolutional ARMA filters[J].arXiv：1901.01343，2019.
[28] XU K，HU W，LESKOVEC J，et al.How powerful are graph neural networks[J].arXiv：1810.00826，2018.
[29] WANG H，ZHAO M，XIE X，et al.Knowledge graph convolutional networks for recommender systems.corr abs/1904.12575（2019）[J].arXiv：1904.12575，2019.
[30] CAI D，LAM W.Graph transformer for graph-to-sequence learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2020：7464-7471.
[31] YANG L，LI W，GUO Y，et al.Graph-CAT：graph co-attention networks via local and global attribute augmentations[J].Future Generation Computer Systems，2021，118：170-179.
[32] CHEN J，CHEN H.Edge-featured graph attention network[J].arXiv：2101.07671.2021.
[33] 唐朝生，胡超超，孙君顶，等.医学图像深度学习技术：从卷积到图卷积的发展[J].中国图象图形学报，2021，26（9）：2078-2093.
TANG C S，HU C C，SUN J D，et al.Deep learning-based medical images analysis evolved from convolution to graph convolution[J].Journal of Image Graphics，2021，26（9）：2078-2093.
[34] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[35] CHIANG W，LIU X，SI S，et al.Cluster-GCN：an efficient algorithm for training deep and large graph convolutional networks[C]//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining，2019：257-266.
[36] TANG J，GAO X，HU W.RGLN：robust residual graph learning networks via similarity-preserving mapping on graphs[C]//2021 IEEE International Conference on Acoustics，Speech and Signal Processing（ICASSP），2021：2940-2944.
[37] YANG H，ZHEN X，CHI Y，et al.Cpr-GCN：conditional partial-residual graph convolutional network in automated anatomical labeling of coronary arteries[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：3803-3811.
[38] ZHOU C.A hybrid approach for coronary artery anatomical labeling in cardiac CT angiography[C]//Journal of Physics：Conference Series，2020.
[39] TIAN Z，LI X，ZHENG Y，et al.Graph-convolutional-network-based interactive prostate segmentation in MR images[J].Medical Physics，2020，47（9）：4164-4176.
[40] ZHAO T，CAO K，YAO J，et al.3D graph anatomy geometry-integrated network for pancreatic mass segmentation，diagnosis，and quantitative patient management[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：13743-13752.
[41] HU K，LIU D，CHEN Z，et al.Embedded residual recurrent network and graph search for the segmentation of retinal layer boundaries in optical coherence tomography[J].IEEE Transactions on Instrumentation and Measurement，2021，70：1-17.
[42] OUYANG S，LI Y.Combining deep semantic segmentation network and graph convolutional neural network for semantic segmentation of remote sensing imagery[J].Remote Sensing，2021，13（1）：119.
[43] MENG Y，ZHANG H，GAO D，et al.BI-GCN：boundary-aware input-dependent graph convolution network for biomedical image segmentation[J].arXiv：2110.14775，2021.
[44] LI G，MüLLER M，QIAN G，et al.Deepgcns：making GCNS go as deep as cnns[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2021.
[45] XU K，BA J，KIROS R，et al.Show，attend and tell：neural image caption generation with visual attention[C]//International Conference on Machine Learning，2015：2048-2057.
[46] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems，2017：5998-6008.
[47] YANG Z，YANG D，DYER C，et al.Hierarchical attention networks for document classification[C]//Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics：Human Language Technologies，2016：1480-1489.
[48] VELI?KOVI? P，CUCURULL G，CASANOVA A，et al.Graph attention networks[J].arXiv：1710.10903，2017.
[49] CUCURULL G，WAGSTYL K，CASANOVA A，et al.Convolutional neural networks for mesh-based parcellation of the cerebral cortex[C]//The International Conference on Medical Imaging with Deep Learning（MIDL），Amsterdam，2018.
[50] HAMPE N，WOLTERINK J M，COLLET C，et al.Graph attention networks for segment labeling in coronary artery trees[C]//Medical Imaging 2021：International Society for Optics and Photonics，2021.
[51] CUI H，XUAN P，JIN Q，et al.Co-graph attention reasoning based imaging and clinical features integration for lymph node metastasis prediction[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention，2021：657-666.
[52] KEICHER M，BURWINKEL H，BANI-HAROUNI D，et al.U-GAT：multimodal graph attention network for COVID-19 outcome prediction[J].arXiv：2108.00860，2021.
[53] HAO J，LIU J，PEREIRA E，et al.Uncertainty-guided graph attention network for parapneumonic effusion diagnosis[J].Medical Image Analysis，2021，75：102217.
[54] YIN S，PENG Q，LI H，et al.Multi-instance deep learning with graph convolutional neural networks for diagnosis of kidney diseases using ultrasound imaging[C]//International Workshop on Uncertainty for Safe Utilization of Machine Learning in Medical Imaging and Clinical Image-Based Procedures，2019：146-154.
[55] CHANG H，ZHAO S，ZHENG H，et al.Multi-vertebrae segmentation from arbitrary spine MR images under global view[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention，2020：702-711.
[56] LU Y，CHEN Y，ZHAO D，et al.CNN-G：convolutional neural network combined with graph for image segmentation with theoretical analysis[J].IEEE Transactions on Cognitive and Developmental Systems，2021，13（3）：631-644.
[57] MA Q，ZHOU S，LI C，et al.Dual graph reasoning unit for brain tumor segmentation[C]//2021 IEEE 4th International Conference on Big Data and Artificial Intelligence（BDAI），2021：204-208.
[58] LI R，HUANG Y，CHEN H，et al.3D graph-connectivity constrained network for hepatic vessel segmentation[J].IEEE Journal of Biomedical and Health Informatics，2021：2168-2194.
[59] CHEN R J，LU M Y，SHABAN M，et al.Whole slide images are 2D point clouds：context-aware survival prediction using patch-based graph convolutional networks[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention，2021：339-349.
[60] JIA H，TANG H，MA G，et al.PSGR：pixel-wise sparse graph reasoning for COVID-19 pneumonia segmentation in CT images[J].arXiv：2108.03809，2021.
[61] WANG M，ZHENG K，YANG Y，et al.An explainable machine learning framework for intrusion detection systems[J].IEEE Access，2020，8：73127-73141.
[62] ZHANG Y，HUANG K，GENG G，et al.Fast kNN graph construction with locality sensitive hashing[C]//Joint European Conference on Machine Learning and Knowledge Discovery in Databases，2013：660-674.
[63] WOLTERINK J M，LEINER T，I?GUM I.Graph convolutional networks for coronary artery segmentation in cardiac CT angiography[C]//International Workshop on Graph Learning in Medical Imaging，2019：62-69.
[64] SUN K，ZHU Z，LIN Z.Adagcn：adaboosting graph convolutional networks into deep models[J].arXiv：1908.
05081.2019.
[65] IVANOV S，PROKHORENKOVA L.Boost then convolve：gradient boosting meets graph neural networks[J].arXiv：2101.08543，2021.
[66] KE G，MENG Q，FINLEY T，et al.Lightgbm：a highly efficient gradient boosting decision tree[J].Advances in Neural Information Processing Systems，2017，30：3146-3154.
[67] SOBERANIS-MUKUL R D，NAVAB N，ALBARQOUNI S.Uncertainty-based graph convolutional networks for organ segmentation refinement[C]//Medical Imaging with Deep Learning，2020：755-769.
[68] DEMIR U，OZER A，SAHIN Y H，et al.Uncertainty-based dynamic graph neighborhoods for medical segmentation[C]//International Workshop on Predictive Intelligence in Medicine，2021：255-265.
[69] WU Z，ZHAO F，XIA J，et al.Intrinsic patch-based cortical anatomical parcellation using graph convolutional neural network on surface manifold[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention，2019：492-500.
[70] LI J，FENG C，LIN X，et al.Utilizing GCN and meta-learning strategy in unsupervised domain adaptation for pancreatic cancer segmentation[J].IEEE Journal of Biomedical and Health Informatics，2021：2168-2194.
[71] GAO R，HOU Z，LI J，et al.Joint coronary centerline extraction and lumen segmentation from ccta using cnntracker and vascular graph convolutional network[C]//2021 IEEE 18th International Symposium on Biomedical Imaging（ISBI），2021：1897-1901.
[72] KUMAR A，TRIPATHI A R，SATAPATHY S C，et al.SARS-Net：COVID-19 detection from chest X-rays by combining graph convolutional network and convolutional neural network[J].Pattern Recognition，2022，122：108255.
[73] GAGGION N，MANSILLA L，MILONE D，et al.Hybrid graph convolutional neural networks for landmark-based anatomical segmentation[J].arXiv：2106.09832，2021.
[74] JOSHI A，SHARMA K K.Hybrid topology of graph convolution and autoencoder deep network for multiple sclerosis lesion segmentation[C]//2021 International Conference on Artificial Intelligence and Smart Systems（ICAIS），2021：1529-1534.
[75] ZHANG J，HUA Z，YAN K，et al.Joint fully convolutional and graph convolutional networks for weakly-supervised segmentation of pathology images[J].Medical Image Analysis，2021，73：102183.
[76] ZHAO K，DUKA B，XIE H，et al.A dynamic graph convolutional neural network framework reveals new insights into connectome dysfunctions in ADHD[J].Neuro Image，2022，246：118774.
[77] KAZI A，SHEKARFOROUSH S，KRISHNA S A，et al.Inceptiongcn：receptive field aware graph convolutional network for disease prediction[C]//International Conference on Information Processing in Medical Imaging，2019：73-85.
[78] CHEN J，MA T，XIAO C.Fastgcn：fast learning with graph convolutional networks via importance sampling[J].arXiv：1801.10247，2018.
[79] CHEN J，ZHU J，SONG L.Stochastic training of graph convolutional networks with variance reduction[J].arXiv：1710.
10568，2017.