Review of Medical Image Segmentation Algorithms Based on U-Net Variants

doi:10.3778/j.issn.1002-8331.2310-0335

Abstract

Abstract: The simple and efficient network structure of U-Net is widely used in medical image segmentation, and many scholars have made various researches on the U-Net structure. This paper elucidates in the following: firstly, the paper summarizes the key challenges of the U-Net network in the field of medical image segmentation; next, it elaborates the formats and characteristics of medical image datasets that are commonly used in the U-Net network; then, it summarizes the six improvement mechanism of U-Net：skip connection mechanism, generative adversarial network, residual connection mechanism, 3D-UNet, Transformer mechanism, and dense connecting mechanism. Finally, the paper discusses the relationship between these improvement mechanisms and commonly used medical data formats, and points out the ideas and directions for future improvement, so as to stimulate the unlimited potential of U-Net in medical image segmentation.

Key words: U-Net variants, medical image segmentation, semantic segmentation, deep learning, improved mechanism

摘要： U-Net简单高效的网络结构，被广泛应用于医学图像分割任务中，学者们针对U-Net结构进行了很多的研究和改进。基于U-Net网络结构的改进方法从以下方面进行归纳总结：总结了U-Net网络在医学图像分割领域的关键挑战；归纳了常用于U-Net网络的医学图像数据集格式及特点；重点总结U-Net和U-Net变体算法六大改进机制：跳跃连接机制、生成对抗网络、残差连接机制、3D-UNet、Transformer机制、密集连接机制。最后，探讨六大改进机制与常用医学数据之间的关系，并指出未来改进思路和方向，激发U-Net在医学图像分割的无限潜力。

关键词: U-Net变体, 医学图像分割, 语义分割, 深度学习, 改进机制

CUI Ke, TIAN Qichuan, LIAN Lu. Review of Medical Image Segmentation Algorithms Based on U-Net Variants[J]. Computer Engineering and Applications, 2024, 60(11): 32-49.

崔珂, 田启川, 廉露. 基于U-Net变体的医学图像分割算法综述[J]. 计算机工程与应用, 2024, 60(11): 32-49.

References

[1] 呼伟, 徐巧枝, 葛湘巍, 等. 医学图像分割的无监督域适应研究综述[J]. 计算机工程与应用, 2024, 60(6): 10-26.
HU W, XU Q Z, GE X W, et al. Review of unsupervised domain adaptation studies for medical image segmentation[J]. Computer Engineering and Applications, 2024, 60(6): 10-26.
[2] JASWAL D, SOWMYA V, SOMAN K P . Image classification using convolutional neural networks[J]. International Journal of entific and Engineering Research, 2014, 5(6): 1661-1668.
[3] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 2015.
[4] LITJENS G, KOOI T, BEJNORDI B E, et al. A survey on deep learning in medical image analysis[J]. Medical Image Analysis, 2017, 42: 60-88.
[5] 苏润. 基于U-Net框架的医学图像分割若干关键问题研究[D]. 合肥: 中国科学技术大学, 2021.
SU R. Research on several key problems in medical image segmentation based on the U-Net framework[D]. Hefei: University of Science and Technology of China, 2021.
[6] MILLETARI F, NAVAB N, AHMADI S A. V-Net: fully convolutional neural networks for volumetric medical image segmentation[C]//Proceedings of the Fourth International Conference on 3D Vision (3DV), Stanford, CA, USA, 2016.
[7] WONG K C L, MORADI M, TANG H, et al. 3D segmentation with exponential logarithmic loss for highly unbalanced object sizes[C]//Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention, 2018: 612-619.
[8] JIANG Z, DING C, LIU M, et al. Two-stage cascaded U-Net: 1st place solution to BraTS challenge 2019 segmentation task[C]//Proceedings of the International MICCAI Brainlesion Workshop, 2020: 231-241.
[9] DOLZ J, GOPINATH K, YUAN J, et al. HyperDense-Net: a hyper-densely connected cnn for multi-modal image segmen-tation[J]. IEEE Transactions on Medical Imaging, 2019, 38(5): 1116-1126.
[10] BILIC P, CHRIST P, LI H B, et al. The liver tumor segmentation benchmark (LITS)[J]. Medical Image Analysis, 2023, 84: 102680.
[11] BAKAS S, REYES M, BATTISTELLA E, et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BraTS challenge[J]. arXiv:1811.02629, 2018.
[12] AL-DHABYANI W, GOMAA M, KHALED H, et al. Dataset of breast ultrasound images[J]. Data in Brief, 2020, 28: 104863.
[13] STAAL J, ABRAMOFF M D, NIEMEIJER M, et al. Ridge-based vessel segmentation in color images of the retina[J]. IEEE Transactions on Medical Imaging, 2004, 23(4): 501-509.
[14] KUMAR N, VERMA R, SHARMA S, et al. A dataset and a technique for generalized nuclear segmentation for computational pathology[J]. IEEE Transactions on Medical Imaging, 2017, 36(7): 1550-1560.
[15] MOREIRA I C, AMARAL I, DOMINGUES I, et al. INbreast: toward a full-field digital mammographic database[J]. Academic Radiology, 2012, 19(2): 236-248.
[16] RONNEBERGER O, FISCHER P, BROX T. U-Net: convo- lutional networks for biomedical image segmentation[J]. arXiv:1505.04597, 2015.
[17] DROZDZAL M, VORONTSOV E, CHARTRAND G, et al. The importance of skip connections in biomedical image segmentation[J]. arXiv:1608.04117, 2016.
[18] HUANG H, LIN L, TONG R, et al. UNet3+: a full-scale connected unet for medical image segmentation[C]//Proceedings of the 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, 2020.
[19] XIANG T, ZHANG C, LIU D, et al. BiO-Net: learning recurrent bi-directional connections for encoder-decoder architecture[C]//Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention, 2020: 74-84.
[20] LIU Y, WANG H, CHEN Z, et al. TransUNet+: redesigning the skip connection to enhance features in medical image segmentation[J]. Knowledge-Based Systems, 2022, 256: 109859.
[21] IBTEHAZ N, RAHMAN M S. MultiResUNet: rethinking the U-Net architecture for multimodal biomedical image segme-ntation[J]. Neural Networks, 2020, 121: 74-87.
[22] LI C, TAN Y, CHEN W, et al. Attention Unet++: a nested attention-aware U-Net for liver CT image segmentation[C]//Proceedings of the 2020 IEEE International Conference on Image Processing (ICIP), Abu Dhabi, United Arab Emirates, 2020.
[23] JIN Q G, MENG Z P, SUN C M, et al. RA-UNet: a hybrid deep attention-aware network to extract liver and tumor in CT scans[J]. arXiv:1811.01328, 2018.
[24] NODIROV J, ABDUSALOMOV A B, WHANGBO T K. Attention 3D U-Net with multiple skip connections for segmentation of brain tumor images[J]. Sensors, 2022, 22(17): 6501.
[25] CHEN Y, ZHANG J, ZONG X, et al. FSC-UNet: a lightweight medical image segmentation algorithm fused with skip connections[C]//Proceedings of the 14th International Conference on Digital Image Processing (ICDIP 2022), Wuhan, China, 2022.
[26] LI H, FANG J, LIU S, et al. CR-Unet: a composite network for ovary and follicle segmentation in ultrasound images[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 24(4): 974-983.
[27] LACHINOV D, SEEB?CK P, MAI J, et al. Projective skip-connections for segmentation along a subset of dimensions in retinal OCT[C]//Proceedings of the 24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021), 2021: 431-441.
[28] AZAD R, ASADI-AGHBOLAGHI M, FATHY M, et al. Bi-directional ConvLSTM U-Net with densley connected convolutions[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision Workshop, 2019.
[29] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Advances in Neural Information Processing Systems, 2014: 2672-2680.
[30] CHEN Y, JAKARY A, AVADIAPPAN S, et al. QSMGAN: improved quantitative susceptibility mapping using 3D generative adversarial networks with increased receptive field[J]. NeuroImage, 2020, 207: 116389.
[31] WU C, ZOU Y, YANG Z. U-GAN: generative adversarial networks with U-Net for retinal vessel segmentation[C]//Proceedings of the 14th International Conference on Computer Science & Education (ICCSE), Toronto, ON, Canada, 2019.
[32] SHEN T, GOU C, WANG F Y, et al. Learning from adversarial medical images for X-ray breast mass segmentation[J]. Computer Methods and Programs in Biomedicine, 2019, 180: 105012.
[33] CAO X, YANG J, ZHANG J, et al. Deformable image registration based on similarity-steered CNN regression[C]//Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2017), 2017: 300-308.
[34] LI G, ZHANG L, HU S, et al. Adversarial network with dual U-Net model and multiresolution loss computation for medical images registration[C]//Proceedings of the 12th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), Suzhou, China, 2019.
[35] SUN Y, YUAN P, SUN Y. MM-GAN: 3D MRI data augmentation for medical image segmentation via generative adversarial networks[C]//Proceedings of the 2020 IEEE International Conference on Knowledge Graph (ICKG), Nanjing, China, 2020.
[36] XU Z W, CAO Y K, JIN C, et al. GASNet: weakly-supervised framework for COVID-19 lesion segmentation[J]. arXiv:2010.
09456, 2020.
[37] BEJI A, BLAIECH A G, SAID M, et al. An innovative medical image synthesis based on dual GAN deep neural networks for improved segmentation quality[J]. Applied Intelligence, 2023, 53(3): 3381-3397.
[38] DAI W, LIU S, ENGSTROM CRAIG B, et al. Explainable semantic medical image segmentation with style[J]. arXiv:2303.0569, 2023.
[39] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016.
[40] DAS S, DEKA A, IWAHORI Y, et al. Contour-aware residual W-Net for nuclei segmentation[J]. Procedia Computer Science, 2019, 159: 1479-1488.
[41] SHUVO Md B, AHOMMED R, REZA S, et al. CNL-UNet: a novel lightweight deep learning architecture for multimodal biomedical image segmentation with false output suppression[J]. Biomedical Signal Processing and Control, 2021, 70: 102959.
[42] YUAN H, LIU Z, SHAO Y, et al. ResD-Unet research and application for pulmonary artery segmentation[J]. IEEE Access, 2021, 9: 67504-67511.
[43] FANG Z, CHEN Y, NIE D, et al. RCA-U-Net: residual channel attention U-Net for fast tissue quantification in magnetic resonance fingerprinting[C]//Proceedings of the 22th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2019), 2019: 101-109.
[44] RAHMAN A A, BISWAL B, PAPPU G P, et al. Robust segmentation of vascular network using deeply cascaded AReN-UNet[J]. Biomedical Signal Processing and Control, 2021, 69: 102953.
[45] LIU Y, QI N, ZHU Q, et al. CR-U-Net: cascaded U-Net with residual mapping for liver segmentation in CT images*[C]//Proceedings of the 2019 IEEE Visual Communications and Image Processing (VCIP), Sydney, NSW, Australia, 2019.
[46] WANG X, WANG Y. Composite attention residual U-Net for rib fracture detection[J]. Entropy, 2023, 25(3): 466.
[47] BALDEON-CALISTO M, LAI-YUEN S K. AdaResU-Net: multiobjective adaptive convolutional neural network for medical image segmentation[J]. Neurocomputing, 2020, 392: 325-340.
[48] ZULFIQAR M, STANUCH M, WODZINSKI M, et al. DRU-Net: pulmonary artery segmentation via dense residual U-Network with hybrid loss function[J]. Sensors, 2023, 23(12): 5427.
[49] SHARMA A, MISHRA P K. DRI-UNet: dense residual-inception u-net for nuclei identification in microscopy cell images[J]. Neural Computing and Applications, 2023, 35: 19187-19220.
[50] LI H, NAN Y, DEL SER J, YANG G. Large-kernel attention for 3D medical image segmentation[J]. arXiv:2207.11225, 2022.
[51] FAN T, WANG G, LI Y, et al. MA-Net: a multi-scale attention network for liver and tumor segmentation[J]. IEEE Access, 2020, 8: 179656-179665.
[52] SHAN T, YAN J, CUI X, et al. DSCA-Net: a depthwise separable convolutional neural network with attention mecha- nism for medical image segmentation[J]. Mathematical Biosciences and Engineering, 2023, 20: 365-382.
[53] SHAKER A, MAAZ M, RASHEED H, et al. UNETR++: delving into efficient and accurate 3D medical image segmentation[J]. arXiv:2212.04497, 2022.
[54] XIE Y, ZHANG J, LIAO Z, et al. PGL: prior-guided local self-supervised learning for 3D medical image segmentation[J]. arXiv:2011.12640, 2020.
[55] SHIN H, KIM H, KIM S, et al. COSMOS: cross-modality unsupervised domain adaptation for 3D medical image segmentation based on target-aware domain translation and iterative self-training[J]. arXiv:2203.16557, 2022.
[56] LIANG B, TANG C, ZHANG W, et al. N-Net: an UNet architecture with dual encoder for medical image segmentation[J]. Signal Image and Video Processing, 2023, 17(6): 3073-3081.
[57] XU G, WU X, ZHANG X, et al. LeViT-UNet: make faster encoders with transformer for medical image segmentation[J]. arXiv:2107.08623, 2021.
[58] HE Y, YANG D, ROTH H, et al. DiNTS: differentiable neural network topology search for 3D medical image segmentation[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021.
[59] XIA T J, HUANG G H, PUN C M, et al. Multi-scale contextual semantic enhancement network for 3D medical image segmentation[J]. Physics in Medicine and Biology, 2022, 67(22): 225014.
[60] GUO D, TERZOPOULOS D. A transformer-based network for anisotropic 3D medical image segmentation[C]//Proceedings of the 25th International Conference on Pattern Recognition (ICPR), Milan, Italy, 2021.
[61] CHEN R, WANG X, JIN B, et al. CLD-Net: complement local detail for medical small-object segmentation[C]//Proceedings of the 2022 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Las Vegas, NV, USA, 2022.
[62] 石磊, 籍庆余, 陈清威, 等. 视觉Transformer在医学图像分析中的应用研究综述[J]. 计算机工程与应用, 2023, 59(8): 41-55.
SHI L, JI Q Y, CHEN Q W, et al. Review of research on application of vision Transformer in medical image analysis[J]. Computer Engineering and Applications, 2023, 59(8): 41-55.
[63] CHEN J, LU Y, YU Q, et al. TransUNet: Transformers make strong encoders for medical image segmentation[J]. arXiv:2102.04306, 2021.
[64] WANG W, CHEN C, DING M, et al. TransBTS: multimodal brain tumor segmentation using transformer[C]//Proceedings of the 24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021), 2021: 109-119.
[65] LI X, FANG X, YANG G, et al. TransU2-Net: an effective medical image segmentation framework based on transformer and U2-Net[J]. IEEE Journal of Translational Engineering in Health and Medicine, 2023(1): 441-450.
[66] WANG H, CAO P, WANG J, et al. UCTransNet: rethinking the skip connections in U-Net from a channel-wise perspective with transformer[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2022: 2441-2449.
[67] LI Y, WANG S, WANG J, et al. GT U-Net: a U-Net like group transformer network for tooth root segmentation[C]// International Workshop on Machine Learning in Medical Imaging, 2021: 386-395.
[68] XIE Y, ZHANG J, SHEN C, et al. CoTr: efficiently bridging CNN and transformer for 3D medical image segmentation[C]//Proceedings of the 24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021), 2021: 171-180.
[69] HATAMIZADEH A, TANG Y, NATH V, et al. UNETR: Transformers for 3D medical image segmentation[C]//Proceedings of the 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2022.
[70] CAO H, WANG Y, CHEN J, et al. Swin-UNet: UNet-like pure transformer for medical image segmentation[C]//Proceedings of the ECCV 2022 Workshops, 2022: 205-218.
[71] HATAMIZADEH A, NATH V, TANG Y, et al. Swin UNETR: swin Transformers for semantic segmentation of brain tumors in MRI images[J]. arXiv:2201.01266, 2022.
[72] WANG H, XIE S, LIN L, et al. Mixed transformer U-Net for medical image segmentation[C]//Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022.
[73] HUANG H, XIE1 S, LIN L, et al. ScaleFormer: revisiting the transformer-based backbones from a scale-wise perspective for medical image segmentation[C]//Proceedings of the 31st International Joint Conference on Artificial Intelligence, Vienna, Austria, 23-29 July, 2022: 964-971.
[74] HATAMIZADEH A, XU Z, YANG D, et al. UNetFormer: a unified vision transformer model and pre-training framework for 3D medical image segmentation[J]. arXiv:2204.00631, 2022.
[75] JIANG Y, ZHANG Y, LIN X, et al. SwinBTS: a method for 3D multimodal brain tumor segmentation using Swin Transformer[J]. Brain Sciences, 2022, 12(6): 797.
[76] HEIDARI M, KAZEROUNI A, SOLTANY M, et al. HiFormer: hierarchical multi-scale representations using Transformers for medical image segmentation[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023: 6191-6201.
[77] VALANARASU J M J, OZA P, HACIHALILOGLU I, et al. Medical transformer: gated axial-attention for medical image segmentation[C]//Proceedings of the 24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021), 2021: 36-46.
[78] HUANG X, DENG Z, LI D, et al. MISSFormer: an effective medical image segmentation transformer[J]. arXiv:2109.
07162, 2021.
[79] WU Y, LIAO K, CHEN J, et al. D-Former: a U-shaped dilated transformer for 3D medical image segmentation[J]. Neural Computing & Applications, 2023, 35(2): 1931-1944.
[80] HUANG G, LIU Z, VAN DER MAATEN L, et al. Densely connected convolutional networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017.
[81] ZHANG Z, WU C, COLEMAN S, et al. DENSE-INception U-Net for medical image segmentation[J]. Computer Methods and Programs in Biomedicine, 2020, 192: 105395.
[82] MENG C, SUN K, GUAN S, et al. Multiscale dense convolutional neural network for DSA cerebrovascular segmentation[J]. Neurocomputing, 2020, 373: 123-134.
[83] QIANG Z, TU S, XU L. A K-Dense-UNet for biomedical image segmentation[C]//Proceedings of the IScIDE 2019, 2019: 552-562.
[84] TRAN S T, CHENG C H, NGUYEN T T, et al. TMD-Unet: triple-unet with multi-scale input features and dense skip connection for medical image segmentation[J]. Healthcare, 2021 9(1): 54.
[85] TRAN S T, NGUYEN T T, LE M H, et al. TDC-Unet: triple Unet with dilated convolution for medical image segmentation[J]. International Journal of Pharma Medicine and Biological Sciences, 2021, 11(1): 1-7.
[86] MUBASHAR M, ALI H, GR?NLUND C, et al. R2U++: a multiscale recurrent residual U-Net with dense skip connections for medical image segmentation[J]. Neural Computing and Applications, 2022: 34(20): 17723-17739.
[87] ZENG Z, FAN C, XIAO L, et al. DEA-UNet: a dense-edge-attention UNet architecture for medical image segmentation[J]. Journal of Electronic Imaging, 2022, 31(4): 043032.
[88] CAI Z, XIN J, SHI P, et al. DSTUNet: UNet with efficient dense swin transformer pathway for medical image segmentation[C]//Proceedings of the 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), Kolkata, India, 2022.
[89] YIN Y, XU W, CHEN L, et al. CoT-UNet++: a medical image segmentation method based on contextual transformer and dense connection[J]. Mathematical Biosciences and Engineering, 2023, 20(5): 8320-8336.
[90] 杨鹤, 柏正尧. CoT-TransUNet: 轻量化的上下文Transformer医学图像分割网络[J]. 计算机工程与应用, 2023, 59(3): 218-225.
YANG H, BAI Z Y. CoT-TransUNet: lightweight context transformer medical image segmentation network[J]. Computer Engineering and Applications, 2023, 59(3): 218-225.
[91] ZHANG G, YANG Z, JIANG S. Automatic lung tumor segmentation from CT images using improved 3D densely connected UNet[J]. Medical & Biological Engineering & Computing, 2022, 60(11): 3311-3323.
[92] WANG Z H, LIU Z, SONG Y Q, et al. Densely connected deep U-Net for abdominal multi-organ segmentation[C]//Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, China, 2019: 1415-1419.
[93] KIRILLOV A, MINTUN E, RAVI N, et al. Segment anything[J]. arXiv:2304.02643, 2023.