Review of Feature Fusion Techniques for Multimodal MRI Brain Tumor Segmentation Methods

doi:10.3778/j.issn.1002-8331.2402-0087

Abstract

Abstract: With the rapid development of artificial intelligence technology, brain tumor segmentation has made significant progress in improving the accuracy and efficiency of clinical diagnosis, research, and treatment. Multimodal medical image segmentation is becoming a current hot topic and trend in brain tumor segmentation research because multimodal medical images can provide more comprehensive and detailed information, enhancing the feasibility and reliability of brain tumor segmentation algorithms. However, the use of multimodal data also faces a series of problems and challenges. Firstly, the features of each modality may not be fully utilized during fusion, leading to information omission and imbalance between modalities. Secondly, the differences between modalities may be too large, causing interference and misleading the performance of the algorithm. Moreover, modality missing is also a common issue, further reducing the effectiveness of the multimodal approaches. Therefore, this paper aims to study the multimodal fusion methods for medical images to help new researchers explore the future directions in multimodal brain tumor segmentation. The focus of this research is to discuss the feature fusion methods of multimodal data in brain tumor segmentation and classify various methods based on the abstraction levels of the fused data. By studying different fusion algorithms and models, it hopes to address the difficulties and challenges in the fusion of multimodal brain tumor data, improve the accuracy and reliability of brain tumor segmentation, and provide better support for the diagnosis and treatment of brain tumors.

Key words: brain tumor segmentation, multi-modal, early fusion, late fusion, multi-level fusion, modal missing

摘要： 随着人工智能技术的快速发展，脑肿瘤分割在提高临床诊断、研究和治疗的准确性和效率方面取得了显著进展。多模态医疗图像分割正在成为当前脑肿瘤分割研究的热点和趋势，这是因为多模态医学图像可以提供更加全面和详尽的信息，增强了脑肿瘤分割算法的可行性和可靠性。然而，多模态数据的使用也面临着一系列的问题和挑战。模态特征在进行融合时可能无法充分利用，导致模态间信息遗漏和不均衡；模态之间的差异过大，可能会产生干扰和误导算法的表现。而且，模态缺失也是一个常见的问题，使多模态方法的效果进一步降低。因此，旨在研究医疗图像多模态融合方法，以帮助新研究者探索未来多模态脑肿瘤分割的方向。该研究的重点在于探讨在脑肿瘤分割中多模态数据的特征融合方法，并根据融合数据的抽象层次对各种方法进行分类。通过研究不同的融合算法和模型，希望能够解决脑肿瘤多模态数据融合的困难和挑战，提高脑肿瘤分割的准确性与可靠性，为脑肿瘤的诊断与治疗提供更好的支持。

关键词: 脑肿瘤分割, 多模态, 早期融合, 晚期融合, 多级别融合, 模态缺失

LIU Haichao, SONG Lijuan. Review of Feature Fusion Techniques for Multimodal MRI Brain Tumor Segmentation Methods[J]. Computer Engineering and Applications, 2024, 60(23): 28-48.

刘海超, 宋丽娟. 多模态MRI脑肿瘤分割方法的特征融合技术综述[J]. 计算机工程与应用, 2024, 60(23): 28-48.

References

[1] BOROLE V Y, NIMBHORE S S, KAWTHEKAR D S S. Image processing techniques for brain tumor detection: a review[J]. International Journal of Emerging Trends & Technology in Computer Science (IJETTCS), 2015, 4(5): 2.
[2] KUMAR S S, MONI R S, RAJEESH J. Automatic segmentation of liver and tumor for CAD of liver[J]. Journal of Advances in Information Technology, 2011, 2(1): 63-70.
[3] GIBBS P, BUCKLEY D L, BLACKBAND S J, et al. Tumour volume determination from MR images by morphological segmentation[J]. Physics in Medicine & Biology, 1996, 41(11): 2437.
[4] LI Q, BAI K X, ZHAO L, et al. Progresss and challenges of MRI brain tumor image segmentation[J]. Journal of Image and Graphics, 2020, 25(3): 419-431.
[5] CHEN L, WU Y, DSOUZA A M, et al. MRI tumor segmentation with densely connected 3D CNN[C]//Proceedings of theMedical Imaging 2018: Image Processing, 2018: 357-364.
[6] RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, October 5-9, 2015. Cham: Springer International Publishing, 2015: 234-241.
[7] ZHOU Z, RAHMAN SIDDIQUEE M M, TAJBAKHSH N, et al. UNet++: a nested U-Net architecture for medical image segmentation[C]//Proceedings of the Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support: 4th International Workshop (DLMIA 2018), and 8th International Workshop (ML-CDS 2018), Held in Conjunction with MICCAI 2018, Granada, Spain, September 20, 2018. Cham: Springer International Publishing, 2018: 3-11.
[8] MILLETARI F, NAVAB N, AHMADI S A. V-Net: fully convolutional neural networks for volumetric medical image segmentation[C]//Proceedings of the 4th International Conference on 3D Vision (3DV), 2016: 565-571.
[9] KAMNITSAS K, FERRANTE E, PARISOT S, et al. DeepMedic for brain tumor segmentation[C]//Proceedings of the Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: Second International Workshop (BrainLes 2016), with the Challenges on BraTS, ISLES and mTOP 2016, Held in Conjunction with MICCAI 2016, Athens, Greece, October 17, 2016. Cham: Springer International Publishing, 2016: 138-149.
[10] LI W, WANG G, FIDON L, et al. On the compactness, efficiency, and representation of 3D convolutional networks: brain parcellation as a pretext task[C]//Proceedings of the 25th International Conference on Information Processing in Medical Imaging, Boone, NC, USA, June 25-30, 2017. Cham: Springer International Publishing, 2017: 348-360.
[11] ISENSEE F, JAEGER P F, KOHL S A A, et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation[J]. Nature Methods, 2021, 18(2): 203-211.
[12] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. Imagenet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems, 2012: 1106-1114.
[13] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv: 1409. 1556, 2014.
[14] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.
[15] IANDOLA F, MOSKEWICZ M, KARAYEV S, et al. DenseNet: implementing efficient convnet descriptor pyramids[J]. arXiv: 1404.1869, 2014.
[16] BADRINARAYANAN V, KENDALL A, CIPOLLA R. SegNet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2481-2495.
[17] ARUNKUMAR N, MOHAMMED M A, ABD GHANI M K, et al. K-means clustering and neural network for object detecting and identifying abnormality of brain tumor[J]. Soft Computing, 2019, 23: 9083-9096.
[18] SRINIVAS B, SASIBHUSHANA R G. Performance evaluation of fuzzy C means segmentation and support vector machine classification for MRI brain tumor[C]//Proceedings of the Soft Computing for Problem Solving(SocProS 2017), Bhubaneswar, India, 2017: 355-367.
[19] MYRONENKO A. 3D MRI brain tumor segmentation using autoencoder regularization[C]//Proceedings of theBrainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 4th International Workshop (BrainLes 2018), Held in Conjunction with MICCAI 2018, Granada, Spain, September 16, 2018. Cham: Springer International Publishing, 2019: 311-320.
[20] AHMAD B, SUN J, YOU Q, et al. Brain tumor classification using a combination of variational autoencoders and generative adversarial networks[J]. Biomedicines, 2022, 10(2): 223.
[21] ALDHAHAB A, IBRAHIM S, MIKHAEL W B. Stacked sparse autoencoder and softmax classifier framework to classify MRI of brain tumor images[J]. International Journal of Intelligent Engineering & Systems, 2020, 13(3): 12.
[22] 刘鹏, 刘伟峰, 唐晓英. 使用深度学习方法的脑肿瘤图像分割综述[J]. 生命科学仪器, 2023, 21(3): 12-18.
LIU P, LIU W F, TANG X Y. A review of brain tumor image segmentation using depth learning[J]. Life Science Instruments, 2023, 21(3): 12-18.
[23] 孙康康, 陈伟, 李奇轩, 等.多模态MRI脑肿瘤图像分割方法研究进展[J].磁共振成像, 2023, 14(11): 164-169.
SUN K K, CHEN W, LI Q X, et al. Research progress of multimodal MRI brain tumor image segmentation methods[J]. Chinese Journal of Magnetic Resonance Imaging, 2023, 14(11): 164-169.
[24] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems, 2017: 5998-6008.
[25] 梁芳烜, 杨锋, 卢丽云, 等.基于卷积神经网络的脑肿瘤分割方法综述[J].计算机工程与应用, 2021, 57(7): 34-43.
LIANG F X, YANG F, LU L Y, et al. Review of brain tumor segmentation methods based on convolutional neural networks[J]. Computer Engineering and Applications, 2021, 57(7): 34-43.
[26] DEVUNOORU S, ALSADOON A, CHANDANA P W C, et al. Deep learning neural networks for medical image segmentation of brain tumours for diagnosis: a recent review and taxonomy[J]. Journal of Ambient Intelligence and Humanized Computing, 2021, 12: 455-483.
[27] DAS S, NAYAK G K, SABA L, et al. An artificial intelligence framework and its bias for brain tumor segmentation: a narrative review[J]. Computers in Biology and Medicine, 2022, 143: 105273.
[28] RANJBARZADEH R, CAPUTO A, TIRKOLAEE E B, et al. Brain tumor segmentation of MRI images: a comprehensive review on the application of artificial intelligence tools[J]. Computers in Biology and Medicine, 2023, 152: 106405.
[29] SHEHAB L H, FAHMY O M, GASSER S M, et al. An efficient brain tumor image segmentation based on deep residual networks (ResNets)[J]. Journal of King Saud University-Engineering Sciences, 2021, 33(6): 404-412.
[30] LUU H M, PARK S H. Extending nnU-Net for brain tumor segmentation[C]//Proceedings of the International MICCAI Brainlesion Workshop. Cham: Springer International Publishing, 2021: 173-186.
[31] MAJI D, SIGEDAR P, SINGH M. Attention Res-UNet with guided decoder for semantic segmentation of brain tumors[J]. Biomedical Signal Processing and Control, 2022, 71: 103077.
[32] ZHANG W, YANG G, HUANG H, et al. ME‐Net: multi‐encoder net framework for brain tumor segmentation[J]. International Journal of Imaging Systems and Technology, 2021, 31(4): 1834-1848.
[33] MARINOV Z, REI? S, KERSTING D, et al. Mirror U-Net: marrying multimodal fission with multi-task learning for semantic segmentation in medical imaging[J]. arXiv:2303. 07126, 2023.
[34] LI Y, JIA F, QIN J. Brain tumor segmentation from multimodal magnetic resonance images via sparse representation[J]. Artificial Intelligence in Medicine, 2016, 73: 1-13.
[35] ZHANG J, ZENG J, QIN P, et al. Brain tumor segmentation of multi-modality MR images via triple intersecting U-Nets[J]. Neurocomputing, 2021, 421: 195-209.
[36] WANG Y, ZHANG Y, HOU F, et al. Modality-pairing learning for brain tumor segmentation[C]//Proceedings of the Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 6th International Workshop (BrainLes 2020), Held in Conjunction with MICCAI 2020, Lima, Peru, October 4, 2020. Cham: Springer International Publishing, 2021: 230-240.
[37] FANG F, YAO Y, ZHOU T, et al. Self-supervised multi-modal hybrid fusion network for brain tumor segmentation[J]. IEEE Journal of Biomedical and Health Informatics, 2021, 26(11): 5310-5320.
[38] AZAD R, KHOSRAVI N, MERHOF D. SMU-Net: style matching U-Net for brain tumor segmentation with missing modalities[C]//Proceedings of the International Conference on Medical Imaging with Deep Learning, 2022: 48-62.
[39] SHEN Y, GAO M. Brain tumor segmentation on MRI with missing modalities[C]//Proceedings of the Information Processing in Medical Imaging: 26th International Conference(IPMI 2019), Hong Kong, China, June 2-7, 2019. Cham: Springer International Publishing, 2019: 417-428.
[40] ZHOU T, CANU S, VERA P, et al. Brain tumor segmentation with missing modalities via latent multi-source correlation representation[C]//Proceedings of the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2020), Lima, Peru, October 4-8, 2020. Cham: Springer International Publishing, 2020: 533-541.
[41] SINGH G, PHOPHALIA A. Multimodal brain tumor segmentation using modified UNet architecture[C]//Proceedings of the International MICCAI Brainlesion Workshop. Cham: Springer International Publishing, 2021: 295-305.
[42] ?I?EK ?, ABDULKADIR A, LIENKAMP S S, et al. 3D U-Net: learning dense volumetric segmentation from sparse annotation[C]//Proceedings of the 19th International Conference on Medical Image Computing and Computer-Assisted Intervention(MICCAI 2016), Athens, Greece, October 17-21, 2016. Cham: Springer International Publishing, 2016: 424-432.
[43] RAHMAN M M, SADIQUE M S, TEMTAM A G, et al. Brain tumor segmentation using UNet-Context encoding network[C]//Proceedings of the International MICCAI Brainlesion Workshop. Cham: Springer International Publishing, 2021: 463-472.
[44] ZHANG H, DANA K, SHI J, et al. Context encoding for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 7151-7160.
[45] WANG W, CHEN C, DING M, et al. TransBTS: multimodal brain tumor segmentation using transformer[C]//Proceedings of the 24th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021), Strasbourg, France, September 27-October 1, 2021. Cham: Springer International Publishing, 2021: 109-119.
[46] XING Y Z, YUAN J, LIU Q, et al. MM-UNet: multi-attention mechanism and multi?scale feature fusion UNet for tumor image segmentation[C]//Proceedings of the 2nd Asia Conference on Algorithms, Computing and Machine Learning, 2023: 253-257.
[47] DING Y, GONG L, ZHANG M, et al. A multi-path adaptive fusion network for multimodal brain tumor segmentation[J]. Neurocomputing, 2020, 412: 19-30.
[48] HU X, LUO W, HU J, et al. Brain SegNet: 3D local refinement network for brain lesion segmentation[J]. BMC Medical Imaging, 2020, 20: 1-10.
[49] SHENG X, ZHANG Y. Multimodal brain tumor image segmentation based on 3D CTrans U-Net: replacing skip connection with channel-wise attention module[C]//Proceedings of the 8th International Conference on Cloud Computing and Big Data Analytics (ICCCBDA), 2023: 538-544.
[50] PEI L, LIU Y. Multimodal brain tumor segmentation using a 3D ResUNet in BraTS 2021[C]//Proceedings of the International MICCAI Brainlesion Workshop. Cham: Springer International Publishing, 2021: 315-323.
[51] 夏景明, 谈玲, 梁颖. LMD-UNet网络在脑肿瘤多模态MRI图像分割中的应用[J].中国生物医学工程学报, 2023, 42(4): 431-441.
XIA J M, TAN L, LIANG Y. Application of LMD-UNet network in multimodal MRI image segmentation of brain tumor[J]. Chinese Journal of Biomedical Engineering, 2023, 42(4): 431-441.
[52] ZHOU X, LI X, HU K, et al. ERV-Net: an efficient 3D residual neural network for brain tumor segmentation[J]. Expert Systems with Applications, 2021, 170: 114566.
[53] ISLAM K T, WIJEWICKREMA S, O’LEARY S. A deep learning framework for segmenting brain tumors using MRI and synthetically generated CT images[J]. Sensors, 2022, 22(2): 523.
[54] 丁熠, 郑伟, 耿技, 等. 基于多层级并行神经网络的多模态脑肿瘤图像分割框架[J].中国图象图形学报, 2023, 28(7): 2182-2194.
DING Y, ZHENG W, GENG J, et al. Multi-level parallel neural networks based multimodal human brain tumor image segmentation framework[J]. Journal of Image and Graphics, 2023, 28(7): 2182-2194.
[55] JIANG Y, ZHANG Y, LIN X, et al. SwinBTS: a method for 3D multimodal brain tumor segmentation using swin transformer[J]. Brain Sciences, 2022, 12(6): 797.
[56] LIU Z, LIN Y, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 10012-10022.
[57] HATAMIZADEH A, NATH V, TANG Y, et al. Swin UNETR: Swin transformers for semantic segmentation of brain tumors in mri images[C]//Proceedings of the International MICCAI Brainlesion Workshop. Cham: Springer International Publishing, 2021: 272-284.
[58] YANG Y, WEI S, ZHANG D, et al. Hierarchical and global modality interaction for brain tumor segmentation[C]//Proceedings of the International MICCAI Brainlesion Workshop. Cham: Springer International Publishing, 2021: 441-450.
[59] SHEN L, ZHANG Y, WANG Q, et al. Feature interaction network based on hierarchical decoupled convolution for 3D medical image segmentation[J]. Plos One, 2023, 18(7): e0288658.
[60] GAO H, MIAO Q, MA D, et al. Deep mutual learning for brain tumor segmentation with the fusion network[J]. Neurocomputing, 2023, 521: 213-220.
[61] HUANG Z, ZHAO Y, LIU Y, et al. GCAUNet: a group cross-channel attention residual UNet for slice based brain tumor segmentation[J]. Biomedical Signal Processing and Control, 2021, 70: 102958.
[62] CHOI Y, AL-MASNI M A, KIM D H. 3D CMM-Net with deeper encoder for semantic segmentation of brain tumors in BraTS2021 challenge[C]//Proceedings of the International MICCAI Brainlesion Workshop. Cham: Springer International Publishing, 2021: 333-343.
[63] LI Z, SHEN Z, WEN J, et al. Automatic brain tumor segmentation using multi-scale features and attention mechanism[C]//Proceedings of the International MICCAI Brainlesion Workshop. Cham: Springer International Publishing, 2021: 216-226.
[64] CHANG Y, ZHENG Z, SUN Y, et al. DPAFNet: a residual dual-path attention-fusion convolutional neural network for multimodal brain tumor segmentation[J]. Biomedical Signal Processing and Control, 2023, 79: 104037.
[65] 张建新, 刘冬伟, 张睦卿, 等.多尺度非局部自注意力MRI脑肿瘤分割网络[J].计算机系统应用, 2024, 33(2): 143-150.
ZHANG J X, LIU D W, ZHANG M Q, et al. MRI brain tumor segmentation network using multi-scale non-local self-attention[J]. Computer Systems and Applications, 2024, 33(2): 143-150.
[66] 宋建丽, 吕晓琪, 谷宇.语义流引导采样结合注意力机制的脑肿瘤图像分割[J].光学精密工程, 2024, 32(4): 565-577.
SONG J L, LYU X Q, GU Y. Brain tumor image segmentation based on semantic flow guided sampling and attention mechanism[J]. Optics and Precision Engineering, 2024, 32(4): 565-577.
[67] WANG Y, CHEN J, BAI X. Gradient-assisted deep model for brain tumor segmentation by multi-modality MRI volumes[J]. Biomedical Signal Processing and Control, 2023, 85: 105066.
[68] WU Q, PEI Y, CHENG Z, et al. SDS-Net: a lightweight 3D convolutional neural network with multi-branch attention for multimodal brain tumor accurate segmentation[J]. Mathematical Biosciences and Engineering, 2023, 20(9): 17384-17406.
[69] YANG H, SHEN Z, LI Z, et al. Combining global information with topological prior for brain tumor segmentation[C]//Proceedings of the International MICCAI Brainlesion Workshop. Cham: Springer International Publishing, 2021: 204-215.
[70] GUAN X, YANG G, YE J, et al. 3D AGSE-VNet: an automatic brain tumor MRI data segmentation framework[J]. BMC Medical Imaging, 2022, 22(1): 1-18.
[71] HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 7132-7141.
[72] 许立君, 黎辉, 刘祖阳, 等. 3D-GA-UNet: 基于3D-Ghost卷积神经网络的脑胶质瘤MRI图像分割算法[J].计算机应用, 2024, 44(4): 1294-1302.
XU L J, LI H, LIU Z Y, et al. 3D-GA-UNet: MRI image segmentation algorithm for glioma based on 3D-Ghost CNN[J].Journal of Computer Applications, 2024, 44(4): 1294-1302.
[73] 李锵, 李瑞璇, 刘恒忻, 等. 循环分层解耦卷积和最大滤波的三维脑肿瘤图像分割[J]. 天津大学学报 (自然科学与工程技术版), 2024, 57(3): 223-233.
LI Q, LI R X, LIU H X, et al. 3D brain tumor image segmentation based on recurrent hierarchical decoupled convolution and maximum filtering[J]. Journal of Tianjin University (Science and Technology), 2024, 57(3): 223-233.
[74] LAN C, YU X, ZHANG L, et al. Brain tumor image segmentation based on improved Polyp-PVT[J]. Signal, Image and Video Processing, 2023, 17(8): 4019-4027.
[75] WANG W, XIE E, LI X, et al. PVT v2: improved baselines with pyramid vision transformer[J]. Computational Visual Media, 2022, 8(3): 415-424.
[76] LIU Y, MU F, SHI Y, et al. SF-Net: a multi-task model for brain tumor segmentation in multimodal MRI via image fusion[J]. IEEE Signal Processing Letters, 2022, 29: 1799-1803.
[77] ZHANG Y, ZHONG P, JIE D, et al. Brain tumor segmentation from multi-modal MR images via ensembling UNets[J]. Frontiers in Radiology, 2021, 1: 704888.
[78] ZHANG Y, WU J, LIU Y, et al. MI-UNet: multi-inputs UNet incorporating brain parcellation for stroke lesion segmentation from T1-weighted magnetic resonance images[J]. IEEE Journal of Biomedical and Health Informatics, 2020, 25(2): 526-535.
[79] YE F, ZHENG Y, YE H, et al. Parallel pathway dense neural network with weighted fusion structure for brain tumor segmentation[J]. Neurocomputing, 2021, 425: 1-11.
[80] 陈柏年, 韩雨童, 何涛, 等.基于级联动态注意力U-Net的脑肿瘤分割方法[J].计算机科学, 2023, 50(S2): 1031-1037.
CHEN B N, HAN Y T, HE T, et al. Cascade dynamic attention U-Net based brain tumor segmentation[J]. Computer Science, 2023, 50(S2): 1031-1037.
[81] HUANG H, YANG G, ZHANG W, et al. A deep multi-task learning framework for brain tumor segmentation[J]. Frontiers in Oncology, 2021, 11: 690244.
[82] KAO P Y, SHAILJA S, JIANG J, et al. Improving patch-based convolutional neural networks for MRI brain tumor segmentation by leveraging location information[J]. Frontiers in Neuroscience, 2020, 13: 1449.
[83] FONOV V S, ROSA-NETO P, COLLINS D L. 3D MRI brain tumour segmentation with autoencoder regularization and hausdorff distance loss function[C]//Proceedings of the International MICCAI Brainlesion Workshop. Cham: Springer International Publishing, 2021: 324-332.
[84] SAUERESSIG C, BERKLEY A, MUNBODH R, et al. A joint graph and image convolution network for automatic brain tumor segmentation[C]//Proceedings of the International MICCAI Brainlesion Workshop. Cham: Springer International Publishing, 2021: 356-365.
[85] FANG L, WANG X. Brain tumor segmentation based on the dual-path network of multi-modal MRI images[J]. Pattern Recognition, 2022, 124: 108434.
[86] LIN T Y, DOLLáR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 2117-2125.
[87] PEIRIS H, CHEN Z, EGAN G, et al. Reciprocal adversarial learning for brain tumor segmentation: a solution to BraTS challenge 2021 segmentation task[C]//Proceedings of the International MICCAI Brainlesion Workshop. Cham: Springer International Publishing, 2021: 171-181.
[88] MIYATO T, MAEDA S, KOYAMA M, et al. Virtual adversarial training: a regularization method for supervised and semi-supervised learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 41(8): 1979-1993.
[89] 张睦卿, 韩雨童, 陈柏年, 等. 基于多尺度伪影生成对抗网络的磁共振成像脑肿瘤分割方法[J].光子学报, 2023, 52(8): 0817002.
ZHANG M Q, HAN Y T, CHEN B N, et al. Magnetic resonance imaging brain tumor segmentation using multiscale ghost generative adversarial network[J]. Acta Photonica Sinica, 2023, 52(8): 0817002.
[90] BAO Z, WANG Y, HUANG J, et al. Multi-modal brain image segmentation based on multi-encoder with hybrid lateral connection (MEHLC-Net)[C]//Proceedings of the 9th International Conference on Biomedical and Bioinformatics Engineering, 2022: 22-28.
[91] ZHOU T, RUAN S, VERA P, et al. A Tri-attention fusion guided multi-modal segmentation network[J]. Pattern Recognition, 2022, 124: 108417.
[92] ZHUANG Y, LIU H, SONG E, et al. A 3D cross-modality feature interaction network with volumetric feature alignment for brain tumor and tissue segmentation[J]. IEEE Journal of Biomedical and Health Informatics, 2022, 27(1): 75-86.
[93] ZHOU T, ZHU S. Uncertainty quantification and attention-aware fusion guided multi-modal MR brain tumor segmentation[J]. Computers in Biology and Medicine, 2023, 163: 107142.
[94] ZHAO C, LIU K, CHEN W, et al. Multi-modality brain tumor segmentation network based on collaborative feature fusion[C]//Proceedings of the 2022 IEEE 17th Conference on Industrial Electronics and Applications, 2022: 1122-1127.
[95] ZHOU T, CANU S, RUAN S. Fusion based on attention mechanism and context constraint for multi-modal brain tumor segmentation[J]. Computerized Medical Imaging and Graphics, 2020, 86: 101811.
[96] LIU Y, MU F, SHI Y, et al. Brain tumor segmentation in multimodal MRI via pixel-level and feature-level image fusion[J]. Frontiers in Neuroscience, 2022, 16: 1000587.
[97] 戴天虹, 宋洁绮. 基于多编码器的多模态MRI脑肿瘤分割[J]. 计算机科学, 2023, 50(S1): 509-514.
DAI T H, SONG J Q. Multimodal MRI brain tumor segmentation based on multi-encoder architecture[J]. Computer Science, 2023, 50(S1): 509-514.
[98] WANG Q, WU B, ZHU P, et al. ECA-Net: efficient channel attention for deep convolutional neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 11534-11542.
[99] ZHANG G, ZHOU J, HE G, et al. Deep fusion of multi-modal features for brain tumor image segmentation[J]. Heliyon, 2023, 9(8).
[100] SYAZWANY N S, NAM J H, LEE S C. MM-BiFPN: multi-modality fusion network with Bi-FPN for MRI brain tumor segmentation[J]. IEEE Access, 2021, 9: 160708-160720.
[101] ZHANG Y, LI Z, LI H, et al. Prototype-driven and multi-expert integrated multi-modal MR brain tumor image segmentation[J]. arXiv:2307.12180, 2023.
[102] ZHENG J, SHI F, ZHAO M, et al. Learning intra-inter-modality complementary for brain tumor segmentation[J]. Multimedia Systems, 2023, 29(6): 3771-3780.
[103] ULLAH F, NADEEM M, ABRAR M, et al. Brain tumor segmentation from MRI images using handcrafted convolutional neural network[J]. Diagnostics, 2023, 13(16): 2650.
[104] SUN Q, FANG N, LIU Z, et al. HybridCTrm: bridging CNN and transformer for multimodal brain image segmentation[J]. Journal of Healthcare Engineering, 2021, 2021: 7467261.
[105] HUANG L, DENOEUX T, VERA P, et al. Evidence fusion with contextual discounting for multi-modality medical image segmentation[C]//Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2022: 401-411.
[106] HAMGHALAM M, LEI B, WANG T. Convolutional 3D to 2D patch conversion for pixel-wise glioma segmentation in MRI scans[C]//Proceedings of the Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries: 5th International Workshop (BrainLes 2019), Held in Conjunction with MICCAI 2019, Shenzhen, China, October 17, 2019. Cham: Springer International Publishing, 2020: 3-12.
[107] ZHANG S, ZHANG J, TIAN B, et al. Multi-modal contrastive mutual learning and pseudo-label re-learning for semi-supervised medical image segmentation[J]. Medical Image Analysis, 2023, 83: 102656.
[108] LIU Y, SHI Y, MU F, et al. Glioma segmentation-oriented multi-modal MR image fusion with adversarial learning[J]. IEEE/CAA Journal of Automatica Sinica, 2022, 9(8): 1528-1531.
[109] LIU Z, CHENG Y, TAN T, et al. MimicNet: mimicking manual delineation of human expert for brain tumor segmentation from multimodal MRIs[J]. Applied Soft Computing, 2023, 143: 110394.
[110] ZHOU T, NOEUVEGLISE A, MODZELEWSKI R, et al. Prediction of brain tumor recurrence location based on multi-modal fusion and nonlinear correlation learning[J]. Computerized Medical Imaging and Graphics, 2023, 106: 102218.
[111] JOYCE J M. Kullback-Leibler divergence[M]//International encyclopedia of statistical science. Berlin Heidelberg: Springer, 2011.
[112] KONG L, ZHANG Y. Multi-modal brain tumor segmentation using cascaded 3D U-Net[C]//Proceedings of the 6th International Conference on Image, Vision and Computing (ICIVC), 2021: 129-133.
[113] ABOUSSALEH I, RIFFI J, FAZAZY K E, et al. Efficient U-Net architecture with multiple encoders and attention mechanism decoders for brain tumor segmentation[J]. Diagnostics, 2023, 13(5): 872.
[114] SANDLER M, HOWARD A, ZHU M, et al. MobileNetV2: inverted residuals and linear bottlenecks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 4510-4520.
[115] LIU H, NIE D, SHEN D, et al. Multimodal brain tumor segmentation using contrastive learning based feature comparison with monomodal normal brain images[C]//Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2022: 118-127.
[116] 刘万军, 姜岚, 曲海成, 等.融合CNN与Transformer的MRI脑肿瘤图像分割[J].智能系统学报, 2024(4): 1007- 1015.
LIU W J, JIANG L, QU H C, et al. MRI brain tumor image segmentation by fusing CNN and Transformer[J]. CAAI Transactions on Intelligent Systems, 2024(4): 1007- 1015.
[117] CHEN Y, CHEN J, WEI D, et al. OctopusNet: a deep learning segmentation network for multi-modal medical images[C]//Proceedings of the Multiscale Multimodal Medical Imaging: First International Workshop (MMMI 2019), Held in Conjunction with MICCAI 2019, Shenzhen, China, October 13, 2019. Cham: Springer International Publishing, 2020: 17-25.
[118] SUN J, PENG Y, GUO Y, et al. Segmentation of the multimodal brain tumor image used the multi-pathway architecture method based on 3D FCN[J]. Neurocomputing, 2021, 423: 34-45.
[119] ZHUANG Y, LIU H, SONG E, et al. APRNet: A 3D anisotropic pyramidal reversible network with multi-modal cross-dimension attention for brain tissue segmentation in MR images[J]. IEEE Journal of Biomedical and Health Informatics, 2021, 26(2): 749-761.
[120] WU B, ZHANG F, XU L, et al. Modality preserving U-Net for segmentation of multimodal medical images[J]. Quantitative Imaging in Medicine and Surgery, 2023, 13(8): 5242.
[121] 张鹏跃, 马巧梅. 跨模态融合的双注意力脑肿瘤分割算法[J].计算机系统应用, 2024, 33(1): 119-126.
ZHANG P Y, MA Q M. Brain tumor segmentation algorithm based on cross-modal fusion dual attention[J]. Computer Systems Applications, 2024, 33(1): 119-126.
[122] 张奕涵, 柏正尧, 尤逸琳, 等.自适应模态融合双编码器MRI脑肿瘤分割网络[J].中国图象图形学报, 2024, 29(3): 768-781.
ZHANG Y H, BAI Z Y, YOU Y L, et al. Adaptive modal fusion dual encoder MRI brain tumor segmentation network[J]. Journal of Image and Graphics, 2024, 29(3): 768-781.
[123] RANJBARZADEH R, BAGHERIAN KASGARI A, JAFARZADEH GHOUSHCHI S, et al. Brain tumor segmentation based on deep learning and an attention mechanism using MRI multi-modalities brain images[J]. Scientific Reports, 2021, 11(1): 10930.
[124] ZHOU T. Modality-level cross-connection and attentional feature fusion based deep neural network for multi-modal brain tumor segmentation[J]. Biomedical Signal Processing and Control, 2023, 81: 104524.
[125] ZHANG R, JIA S, ADAMU M J, et al. HMNet: hierarchical multi-scale brain tumor segmentation network[J]. Journal of Clinical Medicine, 2023, 12(2): 538.
[126] YANG H, ZHOU T, ZHOU Y, et al. Flexible fusion network for multi-modal brain tumor segmentation[J]. IEEE Journal of Biomedical and Health Informatics, 2023, 27(7): l3349-3359.
[127] ZHU Z, HE X, QI G, et al. Brain tumor segmentation based on the fusion of deep semantics and edge information in multimodal MRI[J]. Information Fusion, 2023, 91: 376-387.
[128] LI X, MA S, XU J, et al. TranSiam: aggregating multi-modal visual features with locality for medical image segmentation[J]. Expert Systems with Applications, 2024, 237: 121574.
[129] LIU Z, WEI J, LI R, et al. Learning multi-modal brain tumor segmentation from privileged semi-paired MRI images with curriculum disentanglement learning[J]. Computers in Biology and Medicine, 2023, 159: 106927.
[130] XING Z H, YU L, WAN L, et al. NestedFormer: nested modality-aware transformer for brain tumor segmentation[C]//Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2022: 140-150.
[131] YU W, LUO M, ZHOU P, et al. Metaformer is actually what you need for vision[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 10819-10829.
[132] DING Y, YU X, YANG Y. RFNet: region-aware fusion network for incomplete multi-modal brain tumor segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 3975-3984.
[133] WANG Z, HONG Y. A2FSeg: adaptive multi-modal fusion network for medical image segmentation[C]//Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2023: 673-681.
[134] FENG X, GHIMIRE K, KIM D D, et al. Brain tumor segmentation for multi-modal MRI with missing information[J]. Journal of Digital Imaging, 2023, 36(5): 2075-2087.
[135] KASLIWAL A, SAGARAM S, SRIVASTAVA L, et al. ReFuSeg: regularized multi-modal fusion for precise brain tumour segmentation[J]. arXiv:2308.13883, 2023.
[136] ZHAO Z, YANG H, SUN J. Modality-adaptive feature interaction for brain tumor segmentation with missing modalities[C]//Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2022: 183-192.
[137] ZHU Y, WANG S, HU Y, et al. DRM-VAE: a dual residual multi variational auto-encoder for brain tumor segmentation with missing modalities[C]//Proceedings of the 2021 IEEE 4th International Conference on Electronics and Communication Engineering (ICECE), 2021: 82-86.
[138] ZHOU T. Feature fusion and latent feature learning guided brain tumor segmentation and missing modality recovery network[J]. Pattern Recognition, 2023, 141: 109665.
[139] ZHANG Y, HE N, YANG J, et al. MmFormer: multimodal medical transformer for incomplete multimodal learning of brain tumor segmentation[C]//Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2022: 107-117.
[140] HUANG Z, LIN L, CHENG P, et al. Multi-modal brain tumor segmentation via missing modality synthesis and modality-level attention fusion[J]. arXiv:2203.04586, 2022.
[141] ZHOU T, CANU S, VERA P, et al. Feature-enhanced generation and multi-modality fusion based deep neural network for brain tumor segmentation with missing MR modalities[J]. Neurocomputing, 2021, 466: 102-112.
[142] DIAO Y, LI F, LI Z. Joint learning-based feature reconstruction and enhanced network for incomplete multi-modal brain tumor segmentation[J]. Computers in Biology and Medicine, 2023, 163: 107234.
[143] YANG H, SUN J, XU Z. Learning unified hyper-network for multi-modal MR image synthesis and tumor segmentation with missing modalities[J]. IEEE Transactions on Medical Imaging, 2023, 42(12): 3678-3689.
[144] MENZE B H, JAKAB A, BAUER S, et al. The multimodal brain tumor image segmentation benchmark (BraTS)[J]. IEEE Transactions on Medical Imaging, 2014, 34(10): 1993-2024.
[145] BAKAS S, AKBARI H, SOTIRAS A, et al. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features[J]. Scientific Data, 2017, 4(1): 1-13.
[146] BAKAS S, REYES M, JAKAB A, et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BraTS challenge[J]. arXiv:1811.02629, 2018.