Review on Application of Generative Adversarial Network in Liver Tumor Image Segmentation

doi:10.3778/j.issn.1002-8331.2203-0229

Abstract

Abstract: Due to the complexity and diversity of liver tumor images and the difficulty in obtaining the data set of liver tumor images, the rapid and accurate diagnosis of liver tumor diseases faces many challenges, especially the segmentation of liver tumors is the key research content. Generative adversarial network（GAN） is widely used in medical image processing because of its strong superiority in semi-supervised learning. In order to analyze the present situation and future development of liver tumor image segmentation, the segmentation methods of liver tumor image using GAN are studied, the network structure and derivative model of GAN model are introduced, and the application of generative adversarial network in liver tumor image segmentation is mainly summarized and analyzed, including the segmentation method based on improved network structure, improved GAN method based on generator or discriminator and other improved methods based on GAN. Finally, on the basis of the existing research progress, this paper summarizes the application of GAN in liver tumor image segmentation, discusses the challenges that GAN faces in liver tumor image segmentation, and looks forward to its future development.

Key words: generative adversarial network（GAN）, image segmentation, liver tumor

摘要： 由于肝脏肿瘤图像复杂多样且肝脏肿瘤图像数据集获取困难等问题，快速准确地诊断肝脏肿瘤疾病面临着诸多挑战，尤其是肝脏肿瘤的分割是其中的关键研究内容。生成对抗网络在半监督学习领域具有强大的优越性，因此其在医学图像处理中得到广泛应用。为了分析肝脏肿瘤图像在分割领域的现状以及未来发展，针对应用GAN的肝脏肿瘤图像分割方法进行研究，介绍GAN模型的网络结构与衍生模型，重点总结并分析生成对抗网络在肝脏肿瘤图像分割中的应用，包括基于网络结构改进的GAN方法、基于生成器或判别器改进的GAN方法和基于GAN的其他改进方法。最后在已有的研究进展和基础之上，对GAN在肝脏肿瘤图像分割中的应用进行总结，讨论GAN在肝脏肿瘤图像分割上所面临的挑战，并对其未来发展进行展望。

关键词: 生成对抗网络（GAN）, 图像分割, 肝脏肿瘤

ZHANG Ying, QIU Dawei, LIU Jing. Review on Application of Generative Adversarial Network in Liver Tumor Image Segmentation[J]. Computer Engineering and Applications, 2022, 58(16): 18-30.

张颖, 仇大伟, 刘静. 生成对抗网络在肝脏肿瘤图像分割中的应用综述[J]. 计算机工程与应用, 2022, 58(16): 18-30.

References

[1] 陶雨溪，蔡泽宇，宋腾飞，等.人工智能和影像组学在原发性肝癌中的应用进展[J].中华放射学杂志，2021，55（11）：1222-1225.
TAO Y X，CAI Z Y，SONG T F，et al.The progress and application of artificial intelligence and radiomics on primary liver cancer[J].Chinese Journal of Radiology，2021，55（11）：1222-1225.
[2] MOSTAFA A，ABD ELFATTAH M，FOUAD A，et al.Region growing segmentation with iterative K-means for CT liver images[C]//2015 4th International Conference on Advanced Information Technology and Sensor Application，2015：88-91.
[3] LI B N，CHUI C K，CHANG S，et al.A new unified level set method for semi-automatic liver tumor segmentation on contrast-enhanced CT images[J].Expert Systems with Applications，2012，39（10）：9661-9668.
[4] LECUN Y，BOSER B，DENKER J S，et al.Backpropagation applied to handwritten zip code recognition[J].Neural Computation，1989，1（4）：541-551.
[5] LONG J，SHELHAMER E，DARRELL T.Fully convolutional networks for semantic segmentation[C]//2015 IEEE Conference on Computer Vision and Pattern Recognition，2015：3431-3440.
[6] BEN-COHEN A，DIAMANT I，KLANG E，et al.Fully convolutional network for liver segmentation and lesions detection[M]//Deep learning and data labeling for medical applications.Cham：Springer，2016：77-85.
[7] RONNEBERGER O，FISCHER P，BROX T.U-Net：convolutional networks for biomedical image segmentation[C]//18th International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham：Springer，2015：234-241.
[8] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial nets[C]//Advances in Neural Information Processing Systems 27，2014.
[9] LUC P，COUPRIE C，CHINTALA S，et al.Semantic segmentation using adversarial networks[J].arXiv：1611.08408，2016.
[10] 陈佛计，朱枫，吴清潇，等.生成对抗网络及其在图像生成中的应用研究综述[J].计算机学报，2021，44（2）：347-369.
CHEN F J，ZHU F，WU Q X，et al.A survey about image generation with generative adversarial nets[J].Chinese Journal of Computers，2021，44（2）：347-369.
[11] 宋杰，肖亮，练智超，等.基于深度学习的数字病理图像分割综述与展望[J].软件学报，2021，32（5）：1427-1460.
SONG J，XIAO L，LIAN Z C，et al.Overview and prospect of deep learning for image segmentation in digital pathology[J].Journal of Software，2021，32（5）：1427-1460.
[12] 李祥霞，谢娴，李彬，等.生成对抗网络在医学图像处理中的应用[J].计算机工程与应用，2021，57（18）：24-37.
LI X X，XIE X，LI B，et al.Application of generative adversarial networks in medical image processing[J].Computer Engineering and Applications，2021，57（18）：24-37.
[13] 曹仰杰，贾丽丽，陈永霞，等.生成式对抗网络及其计算机视觉应用研究综述[J].中国图象图形学报，2018，23（10）：1433-1449.
CAO Y J，JIA L L，CHEN Y X，et al.Review of computer vision based on generative adversarial networks[J].Journal of Image and Graphics，2018，23（10）：1433-1449.
[14] 朱秀昌，唐贵进.生成对抗网络图像处理综述[J].南京邮电大学学报（自然科学版），2019，39（3）：1-12.
ZHU X C，TANG G J.A survey on generative adversarial networks in image processing[J].Journal of Nanjing University of Posts and Telecommunications（Natural Science Edition），2019，39（3）：1-12.
[15] MIRZA M，OSINDERO S.Conditional generative adversarial nets[J].arXiv：1411.1784，2014.
[16] ARJOVSKY M，CHINTALA S，BOTTOU L.Wasserstein generative adversarial networks[C]//International Conference on Machine Learning，2017：214-223.
[17] GULRAJANI I，AHMED F，ARJOVSKY M，et al.Improved training of Wasserstein GANs[J].arXiv：1704.00028，2017.
[18] ISOLA P，ZHU J Y，ZHOU T，et al.Image-to-image translation with conditional adversarial networks[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition，2017：1125-1134.
[19] ZHAO J，MATHIEU M，LECUN Y.Energy-based generative adversarial network[J].arXiv：1609.03126，2016.
[20] ODENA A.Semi-supervised learning with generative adversarial networks[J].arXiv：1606.01583，2016.
[21] 邹秀芳，朱定局.生成对抗网络研究综述[J].计算机系统应用，2019，28（11）：1-9.
ZOU X F，ZHU D J.Review on generative confrontation network[J].Computer Systems & Applications，2019，28（11）：1-9.
[22] 梁俊杰，韦舰晶，蒋正锋.生成对抗网络GAN综述[J].计算机科学与探索，2020，14（1）：1-17.
LIANG J J，WEI J J，JIANG Z F.Generative adversarial networks GAN overview[J].Journal of Frontiers of Computer Science and Technology，2020，14（1）：1-17.
[23] ZHU J Y，PARK T，ISOLA P，et al.Unpaired image-to-image translation using cycle-consistent adversarial networks[C]//2017 IEEE International Conference on Computer Vision，2017：2223-2232.
[24] RADFORD A，METZ L，CHINTALA S.Unsupervised representation learning with deep convolutional generative adversarial networks[J].arXiv：1511.06434，2015.
[25] LEDIG C，THEIS L，HUSZáR F，et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition，2017：4681-4690.
[26] WANG X，YU K，WU S，et al.ESRGAN：enhanced super-resolution generative adversarial networks[C]//15th European Conference on Computer Vision Workshops，2018.
[27] 吴少乾，李西明.生成对抗网络的研究进展综述[J].计算机科学与探索，2020，14（3）：377-388.
WU S Q，LI X M.Survey on research progress of generating adversarial networks[J].Journal of Frontiers of Computer Science and Technology，2020，14（3）：377-388.
[28] 马金林，邓媛媛，马自萍.肝脏肿瘤CT图像深度学习分割方法综述[J].中国图象图形学报，2020，25（10）：2024-2046.
MA J L，DENG Y Y，MA Z P.Review of deep learning segmentation methods for CT images of liver tumors[J].Journal of Image and Graphics，2020，25（10）：2024-2046.
[29] XIA K，YIN H，QIAN P，et al.Liver semantic segmentation algorithm based on improved deep adversarial networks in combination of weighted loss function on abdominal CT images[J].IEEE Access，2019，7：96349-96358.
[30] REZAEI M，YANG H，MEINEL C.Conditional generative refinement adversarial networks for unbalanced medical image semantic segmentation[J].arXiv：1810.03871，2018.
[31] 郑寒.深度学习与解剖学先验融合的医学图像分割研究[D].杭州：浙江大学，2020.
ZHENG H.Medical image segmentation based on fusing deep learning and anatomical prior[D].Hangzhou：Zhejiang University，2020.
[32] 赵建峰.基于生成对抗网络的肝血管瘤和肝细胞癌的增强及检测方法的研究[D].济南：山东师范大学，2020.
ZHAO J F.Research on enhancement and detection of hemangioma and hepatocellular carcinoma based on generative adversarial network[D].Jinan：Shandong Normal University，2020.
[33] TANG Y，CAI J，LU L，et al.CT image enhancement using stacked generative adversarial networks and transfer learning for lesion segmentation improvement[C]//International Workshop on Machine Learning in Medical Imaging.Cham：Springer，2018：46-54.
[34] 陈英，郑铖，易珍，等.肝脏及肿瘤图像分割方法综述[J].计算机应用研究，2022，39（3）：641-650.
CHEN Y，ZHENG C，YI Z，et al.Rview of liver and tumor image segmentation methods[J].Application Research of Computers，2022，39（3）：641-650.
[35] WEI X，CHEN X，LAI C，et al.Automatic liver segmentation in CT images with enhanced GAN and mask region-based CNN architectures[J].BioMed Research International，2021：9956983.
[36] YANG D，XU D G，ZHOU S K，et al.Automatic liver segmentation using adversarial image-to-image network：US10600185B2[P].Siemens Healthcare GmbH，2020.
[37] 张泽林，李宝明，徐军.基于条件生成对抗网络的三维肝脏及肿瘤区域自动分割[J].生物医学工程学志，2021，38（1）：80-88.
ZHANG Z L，LI B M，XU J.Automatic three-dimensional segmentation of liver and tumors regions based on conditional generative adversarial networks[J].Journal of Biomedical Engineering，2021，38（1）：80-88.
[38] HUO Y，XU Z，BAO S，et al.Splenomegaly segmentation using global convolutional kernels and conditional generative adversarial networks[C]//Medical Imaging 2018：Image Processing.International Society for Optics and Photonics，2018：1057409.
[39] PENG C，ZHANG X，YU G，et al.Large kernel matters-improve semantic segmentation by global convolutional network[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition，2017：4353-4361.
[40] CHEN J，YANG L，ZHANG Y，et al.Combining fully convolutional and recurrent neural networks for 3D biomedical image segmentation[C]//Advances in Neural Information Processing Systems，2016：3036-3044.
[41] MA J，DENG Y，MA Z，et al.A liver segmentation method based on the fusion of VNet and WGAN[J].Computational and Mathematical Methods in Medicine，2021：5536903.
[42] HE R，XU S，LIU Y，et al.Three-dimensional liver image segmentation using generative adversarial networks based on feature restoration[J].Frontiers in Medicine，2021，8.
[43] 曹祺炜，王峰，牛锦.基于3D卷积神经网络的脑肿瘤医学图像分割优化[J].现代电子技术，2020，43（3）：4-10.
CAO Q W，WANG F，NIU J.Optimization of brain tumor medical image segmentation based on 3D convolutional neural network[J].Modern Electronis Technique，2020，43（3）：4-10.
[44] KIM B，YE J C.Cycle-consistent adversarial network with polyphase U-Nets for liver lesion segmentation[J].Medical Imaging with Deep Learning，2018.
[45] CHEN L，SONG H，WANG C，et al.Liver tumor segmentation in CT volumes using an adversarial densely connected network[J].BMC Bioinformatics，2019，20（16）：587.
[46] WANG C，SONG H，CHEN L，et al.Automatic liver segmentation using multi-plane integrated fully convolutional neural networks[C]//2018 IEEE International Conference on Bioinformatics and Biomedicine，2018：1-6.
[47] XIAO X，ZHAO J，QIANG Y，et al.Radiomics-guided GAN for segmentation of liver tumor without contrast agents[C]//22nd International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham：Springer，2019：237-245.
[48] 武坤.基于生成对抗网络的肝癌MRI多对比度合成和分割方法研究[D].太原：太原理工大学，2020.
WU K.A Multi-contrast MRI images synthesis and segmentation methods research for hepatocellular carcinoma based on generative adversarial network[D].Taiyuan：Taiyuan University of Technology，2020.
[49] REZAEI M，N?PPI J J，LIPPERT C，et al.Generative multi-adversarial network for striking the right balance in abdominal image segmentation[J].International Journal of Computer Assisted Radiology and Surgery，2020，15（11）：1847-1858.
[50] 闫谙，王卫卫.基于条件能量对抗网络的肝脏和肝肿瘤分割[J].计算机工程与应用，2021，57（11）：179-184.
YAN A，WANG W W.Segmentation of liver and liver tumor based on conditional energy-based GAN[J].Computer Engineering and Applications，2021，57（11）：179-184.
[51] 邓鸿，邓雅心，丁廷波，等.基于生成对抗网络的肝脏CT图像分割[J].北京生物医学工程，2021，40（4）：367-376.
DENG H，DENG Y X，DING T B，et al.Liver CT image segmentation based on generated adversarial network[J].Beijing Biomedical Engineering，2021，40（4）：367-376.
[52] ENOKIYA Y，IWAMOTO Y，CHEN Y W，et al.Automatic liver segmentation using U-Net with Wasserstein GANs[J].Journal of Image and Graphics，2018，7：94-101.
[53] 孟琭，钟健平，李楠.基于GAN的医学图像仿真数据集生成算法[J].东北大学学报（自然科学版），2020，41（3）：332-336.
MENG L，ZHONG J P，LI N.Generating algorithm of medical image simulation data sets based on GAN[J].Journal of Northeastern University（Natural Science），2020，41（3）：332-336.
[54] LIU Y，MENG L，ZHONG J.MAGAN：mask attention generative adversarial network for liver tumor CT image synthesis[J].Journal of Healthcare Engineering，2021：6675259.
[55] BEN-COHEN A，DIAMANT I，KLANG E，et al.Fully convolutional network for liver segmentation and lesions detection[M]//Deep learning and data labeling for medical applications.Cham：Springer，2016：77-85.
[56] BEN-COHEN A，KLANG E，RASKIN S P，et al.Virtual PET images from CT data using deep convolutional networks：initial results[C]//International Workshop on Simulation and Synthesis in Medical Imaging.Cham：Springer，2017：49-57.
[57] BEN-COHEN A，KLANG E，RASKIN S P，et al.Cross-modality synthesis from CT to PET using FCN and GAN networks for improved automated lesion detection[J].Engineering Applications of Artificial Intelligence，2019，78：186-194.
[58] YANG J，DVORNEK N C，ZHANG F，et al.Unsupervised domain adaptation via disentangled representations：application to cross-modality liver segmentation[C]//22nd International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham：Springer，2019：255-263.
[59] TANNER C，OZDEMIR F，PROFANTER R，et al.Generative adversarial networks for MR-CT deformable image registration[J].arXiv：1807.07349，2018.
[60] 张欢，刘静，冯毅博，等.U-Net及其在肝脏和肝脏肿瘤分割中的应用综述[J].计算机工程与应用，2022，58（2）：1-14.
ZHANG H，LIU J，FENG Y B，et al.Review of U-Net and its application in segmentation of liver and liver tumors[J].Computer Engineering and Applications，2022，58（2）：1-14.
[61] 宋姝洁，崔振超，陈丽萍，等.多特征融合神经网络的眼底血管分割算法[J].计算机科学与探索，2021，15（12）：2401-2412.
SONG S J，CUI Z C，CHEN L P，et al.Fundus vessel segmentation algorithm based on multi-feature fusion neural network[J].Journal of Frontiers of Computer Science and Technology，2021，15（12）：2401-2412.