Review of Application of Transfer Learning in Medical Image Analysis

doi:10.3778/j.issn.1002-8331.2106-0103

Abstract

Abstract:

Transfer learning is a new learning paradigm in machine learning. It can overcome the defect that deep learning needs a large number of samples, and can solve the problem of inaccurate model caused by small data set in medical image analysis. Therefore, it has become a research hot spot in the field of medical image analysis after deep learning. Firstly, this paper briefly describes the transfer learning, and then according to the main transfer learning methods currently applied in medical image analysis, namely data-based transfer learning, model-based transfer learning, adversarial transfer learning and mixed transfer learning, sorts out and summarizes the important literature in the field of medical image analysis, and analyzes the mechanism, scope of application, application, advantages and disadvantages of each transfer learning method. Then, the paper summarizes, analyzes and compares these transfer learning methods. Finally, the paper points out the development trend of the research in this field according to the deficiency of the research status, and provides a reference for the further research of transfer learning in this field.

Key words: transfer learning, medical images, machine learning, learning paradigm

摘要：

迁移学习是机器学习中一种新的学习范式，它可以克服深度学习需要大量样本的缺陷，能够解决医学图像分析中数据集较小导致模型不准确的问题，因而成为继深度学习之后在医学图像分析领域的研究热点。对迁移学习进行概要阐述，按照目前医学图像分析中应用的主要迁移学习方法，即基于数据的迁移学习、基于模型的迁移学习、对抗式迁移学习和混合迁移学习，对医学图像分析领域的重要文献进行整理和归纳，分析每种迁移学习的机制、适用范围、应用情况和优缺点，再对这几种迁移学习方法进行总结、分析及比较。针对研究现状的不足指出该领域的研究发展趋势，为迁移学习在该领域的深入研究提供参考。

关键词: 迁移学习, 医学图像, 机器学习, 学习范式

LI Ying. Review of Application of Transfer Learning in Medical Image Analysis[J]. Computer Engineering and Applications, 2021, 57(20): 42-52.

黎英. 迁移学习在医学图像分析中的应用研究综述[J]. 计算机工程与应用, 2021, 57(20): 42-52.

References

[1] 杨强，张宇，戴文渊，等.迁移学习[M].北京：机械工业出版社，2020：19-41，82-86，146-147.
YANG Q，ZHANG Y，DAI W Y，et al.Transfer learning[M].Beijing：China Machine Press，2020：19-41，82-86，146-147.
[2] 李猛，李艳玲，林民.命名实体识别的迁移学习研究综述[J].计算机科学与探索，2021，15（2）：206-218.
LI M，LI Y L，LIN M.Review of transfer learning for named entity recognition[J].Journal of Frontiers of Computer Science and Technology，2021，15（2）：206-218.
[3] 岳帅旭，雷文平，薛阳，等.深度对抗迁移学习的故障诊断方法研究[J/OL].机械科学与技术[2021-04-15].https：//doi.org/10.13433/j.cnki.1003-8728.20200373.
YUE S X，LEI W P，XUE Y，et al.Research on fault diagnosis method of deep adversarial transfer learning[J/OL].Mechanical Science and Technology for Aerospace Engineering[2021-04-15].http：//dio.org/10.13433/j.cnki.1003-8728.
202000373.
[4] LIAO X，QIAN Y，CHEN Y，et al.MMTLNet：multi-modality transfer learning network with adversarial training for 3D whole heart segmentation[J].Comput Med Imaging Graph，2020，85：101785.
[5] WEI Ying，ZHANG Yu，HUANG Junzhou，et al.Transfer learning via learning to transfer[C]//Proceedings of the 35th International Conference on Machine Learning，2018：5072-5081.
[6] CHENG B，LIU M，SUK H I，et al.Multimodal manifold-regularized transfer learning for MCI conversion prediction[J].Brain Imaging Behav，2015，9（4）：913-926.
[7] VAN OPBROEK A，ACHTERBERG H C，VERNOOIJ M W，et al.Transfer learning for image segmentation by combining image weighting and kernel learning[J].IEEE Trans Med Imaging，2019，38（1）：213-224.
[8] VAN ENGELEN A，VAN DIJK A C，TRUIJMAN M T，et al.Multi-center MRI carotid plaque component segmentation using feature normalization and transfer learning[J].IEEE Trans Med Imaging，2015，34（6）：1294-1305.
[9] YANG Y，LI X，WANG P，et al.Multi-source transfer learning via ensemble approach for initial diagnosis of Alzheimer’s disease[J].IEEE J Transl Eng Health Med，2020，8：1400310.
[10] BERMUDEZ C，REMEDIOS S W，RAMADASS K，et al.Generalizing deep whole-brain segmentation for post-contrast MRI with transfer learning[J].J Med Imaging（Bellingham），2020，7（6）：064004.
[11] YAO Y，CUI Z.An automatic epilepsy detection method based on improved inductive transfer learning[J].Comput Math Methods Med，2020，2020：5046315.
[12] CHENG B，ZHANG D，SHEN D.Domain transfer learning for MCI conversion prediction[J].Med Image Comput Comput Assist Interv，2012，15：82-90.
[13] VAN OPBROEK A，ACHTERBERG H C，VERNOOIJ M W，et al.Transfer learning by feature-space transformation：a method for Hippocampus segmentation across scanners[J].Neuroimage Clin，2018，20：466-475.
[14] ZHANG Y，AN M.Deep learning-and transfer learning-based super resolution reconstruction from single medical image[J].J Healthc Eng，2017：5859727.
[15] PEI Z，CAO S，LU L，et al.Direct cellularity estimation on breast cancer histopathology images using transfer learning[J].Comput Math Methods Med，2019：3041250.
[16] ZHENG Qiang，FURTH S L，TASIAN G E，et al.Computer aided diagnosis of congenital abnormalities of the kidney and urinary tract in children based on ultrasound imaging data by integrating texture image features and deep transfer learning image features[J].J Pediatr Urol，2019，15（1）：1-7.
[17] 褚晶辉，吴泽蕤，吕卫，等.基于迁移学习和深度卷积神经网络的乳腺肿瘤诊断系统[J].激光与光电子学进展，2018（8）：202-208.
ZHE J H，WU Z R，LV W，et al.Breast cancer diagnosis system based on transfer learning and deep convolutional neural networks[J].Laser & Optoelectronics Progress，2018（8）：202-208.
[18] DENIZ E，?ENGüR A，KADIRO?LU Z，et al.Transfer learning based histopathologic image classifcation for breast cancer detection[J].Health Inf Sci Syst，2018，6：18.
[19] BURLINA P，PACHECOD K D，JOSHI N，et al.Comparing humans and deep learning performance for grading AMD：a study in using universal deep features and transfer learning for automated AMD analysis[J].Comput Biol Med，2017，82：80-86.
[20] TRIVIZAKIS E，IOANNIDIS G S，MELISSIANOS V D，et al.A novel deep learning architecture outperforming 'off?the?shelf' transfer learning and feature?based methods in the automated assessment of mammographic breast density[J].Oncol Rep，2019，42（5）：2009-2015.
[21] BYRA M，STYCZYNSKI G，SZMIGIELSKI C，et al.Transfer learning with deep convolutional neural network for liver steatosis assessment in ultrasound images[J].International Journal of Computer Assisted Radiology and Surgery，2018（13）：1895-1903.
[22] HUYNH B Q，LI Hui，GIGER M L.Digital mammographic tumor classification using transfer learning from deep convolutional neural networks[J].J Med Imag，2016，3（3）：034501.
[23] PAUL R，HAWKINS S H，BALAGURUNATHAN Y，et al.Deep feature transfer learning in combination with traditional features predicts survival among patients with lung adenocarcinoma[J].Tomography，2016，2（4）：388-395.
[24] ZHANG J，CUI W，GUO X，et al.Classification of digital pathological images of non-Hodgkin’s lymphoma subtypes based on the fusion of transfer learning and principal component analysis[J].Med Phys，2020，47（9）：4241-4253.
[25] YU J，DENG Y，LIU T，et al.Lymph node metastasis prediction of papillary thyroid carcinoma based on transfer learning radiomics[J].Nat Commun，2020，11（1）：4807.
[26] HAN F，YAN L，CHEN J，et al.Predicting unnecessary nodule biopsies from a small，unbalanced，and pathologically proven dataset by transfer learning[J].J Digit Imaging，2020，33（3）：685-696.
[27] OH K，CHUNG Y C，KIM K W，et al.Classifcation and visualization of Alzheimer’s disease using volumetric convolutional neural network and transfer learning[R].Scientific Reports，2019.
[28] LI H，PARIKH N A，HE L.A novel transfer learning approach to enhance deep neural network classification of brain functional connectomes[J].Frontiers in Neuroscience，2018.
[29] DAWUD A M，YURTKAN K，OZTOPRAK H.Application of deep learning in neuroradiology：brain haemorrhage classification using transfer learning[J].Computational Intelligence and Neuroscience，2019（6）：1-12.
[30] 程波，朱丙丽，熊江.基于多模态多标记迁移学习的早期阿尔茨海默病诊断[J].计算机应用，2016，36（8）：2282-2286.
CHENG B，ZHU B L，XIONG J.Multimodal multi-label transfer learning for early diagnosis of Alzheimer’s disease[J].Journal of Computer Applications，2016，36（8）：2282-2286.
[31] DAS N N，KUMAR N，KAUR M，et al.Automated deep transfer learning-based approach for detection of COVID-19 infection in chest X-rays[J].Ing Rech Biomed，2020.
[32] ZHANG R，GUO Z，SUN Y，et al.COVID19XrayNet：a two-step transfer learning model for the COVID-19 detecting problem based on a limited number of chest X-Ray images[J].Interdiscip Sci，2020：1-11.
[33] SAMALA R K，CHAN H P，HADJIISKI L M，et al.Multi-task transfer learning deep convolutional neural network：application to computer-aided diagnosis of breast cancer on mammograms[J].Phys Med Biol，2018，62（23）：8894-8908.
[34] WANG Shudong，DONG Liyuan，WANG Xun，et al.Classification of pathological types of lung cancer from CT images by deep residual neural networks with transfer learning strategy[J].Open Med，2020，15：190-197.
[35] 黄鸿，彭超，吴若愚，等.基于部分注释CT图像的自监督迁移学习肺结节分类[J].光学学报，2020，40（18）：99-106.
HUANG H，PENG C，WU R Y，et al.Self-supervised transfer learning of pulmonary nodule classification based on partially annotated CT images[J].Acta Optica Sinica，2020，40（18）：99-106.
[36] NISHIO M，SUGIYAMA O，YAKAMI M，et al.Computer-aided diagnosis of lung nodule classification between benign nodule，primary lung cancer，and metastatic lung cancer at different image size using deep convolutional neural network with transfer learning[J].PLoS One，2018，13（7）：e0200721.
[37] MOHAMMADI R，SALEHI M，GHAFFARI H，et al.Transfer learning-based automatic detection of coronavirus disease 2019（COVID-19） from chest X-ray images[J].J Biomed Phys Eng，2020，10（5）：559-568.
[38] MAQSOOD M，NAZIR F，KHAN U，et al.Transfer learning assisted classification and detection of Alzheimer’s disease stages using 3D MRI scans[J].Sensors，2019，19：2645.
[39] SI X，ZHANG X，ZHOU Y，et al.Automated detection of juvenile myoclonic epilepsy using CNN based transfer learning in diffusion MRI[C]//Annu Int Conf IEEE Eng Med Biol Soc，2020：1679-1682.
[40] WANG S H，XIE S，CHEN X，et al.Alcoholism identification based on an AlexNet transfer learning model[J].Front Psychiatry，2019，10：205.
[41] TAN T，LI Z，LIU H，et al.Optimize transfer learning for lung diseases in bronchoscopy using a new concept：sequential fine-tuning[J].Medical Imaging and Diagnostic Radiology，2018，6.
[42] HASHMI M F，KATIYAR S，KESKAR A G，et al.Efficient pneumonia detection in chest xray images using deep transfer learning[J].Diagnostics（Basel），2020，10（6）：417.
[43] ROMERO M，INTERIAN Y，SOLBERG T，et al.Targeted transfer learning to improve performance in small medical physics datasets[J].Med Phys，2020，47（12）：6246-6256.
[44] 张驰名，王庆凤，刘志勤，等.基于深度迁移学习的肺结节辅助诊断方法[J].计算机工程，2020，46（1）：271-278.
ZHANG C M，WANG Q F，LIU Z Q，et al.Pulmonary nodule auxiliary diagnosis method based on deep transfer learning[J].Computer Engineering，2020，46（1）：271-278.
[45] YANG Y，YAN L F，ZHANG X，et al.Glioma grading on conventional MR images：a deep learning study with transfer learning[J].Front Neurosci，2018，12：804.
[46] SAMALA R K，CHAN H P，HADJIISKI L，et al.Breast cancer diagnosis in digital breast tomosynthesis：effects of training sample size on multi-stage transfer learning using deep neural nets[J].IEEE Trans Med Imaging，2019，38（3）：686-696.
[47] HOSNY K M，KASSEM M A，FOAUD M M.Classification of skin lesions using transfer learning and augmentation with Alex-net[J].PLoS One，2019，14（5）：e0217293.
[48] KARRI S P K，CHAKRABORTY D，CHATTERJEE J.Transfer learning based classification of optical coherence tomography images with diabetic macular edema and dry age-related macular degeneration[J].Biomedical Optics Express，2017，8（2）：579-592.
[49] CHEN Quan，HU Shiliang，LONG Peiran，et al.A transfer learning approach for malignant prostate lesion detection on multiparametric MRI[J].Technology in Cancer Research & Treatment，2019，18：1-9.
[50] MINAEE S，KAFIEH R，SONKA M，et al.Deep-COVID：predicting COVID-19 from chest X-ray images using deep transfer learning[J].Med Image Anal，2020，65：101794.
[51] 陈道争，江倩.基于卷积神经网络和迁移学习的肺结节检测[J].计算机工程与设计，2021，42（1）：240-247.
CHEN D Z，JIANG Q.Pulmonary nodule detection based on convolutional neural networks with transfer learning[J].Computer Engineering and Design，2021，42（1）：240-247.
[52] 吴英，罗良平，许波，等.基于迁移学习的乳腺肿瘤超声图像智能分类诊断[J].中国医学影像技术，2019，35（3）：357-360.
WU Y，LUO L P，XU B，et al.Classification and diagnosis of ultrasound images with breast tumors based on transfer learning[J].Chin J Med Imaging Technol，2019，35（3）：357-360.
[53] HUANG C，LV W，ZHOU C，et al.Discrimination between transient and persistent subsolid pulmonary nodules on baseline CT using deep transfer learning[J].Eur Radiol，2020，30（12）：6913-6923.
[54] LIU W，CHENG Y，LIU Z，et al.Preoperative prediction of Ki-67 status in breast cancer with multiparametric MRI using transfer learning[J].Acad Radiol，2020，28（2）：44-53.
[55] ABBASI A A，HUSSAIN L，AWAN I A，et al.Detecting prostate cancer using deep learning convolution neural network with transfer learning approach[J].Cogn Neurodyn，2020，14（4）：523-533.
[56] WEE C Y，LIU C，LEE A，et al.Cortical graph neural network for AD and MCI diagnosis and transfer learning across populations[J].NeuroImage：Clinical，2019，23：101929.
[57] 金祝新，秦飞巍，方美娥.深度迁移学习辅助的阿尔兹海默氏症早期诊断[J].计算机应用与软件，2019，36（5）：171-177.
JIN Z X，QIN F W，FANG M E.Deep transfer learning-assisted early diagnosis of Alzheimer’s disease[J].Computer Application and Software，2019，36（5）：171-177.
[58] LEE Y S，JUNG S K，RYU J J，et al.Evaluation of transfer learning with deep convolutional neural networks for screening osteoporosis in dental panoramic radiographs[J].Journal of Clinical Medicine，2020，9：392.
[59] KIM J E，NAM N E，SHIM J S，et al.Transfer learning via deep neural networks for implant fixture system classification using periapical radiographs[J].Journal of Clinical Medicine，2020，9：1117.
[60] PATHAK Y，SHUKLA P K，TIWARI A，et al.Deep transfer learning based classification model for COVID-19 disease[J].Ing Rech Biomed，2020.
[61] LIANG G，ZHENG L.A transfer learning method with deep residual network for pediatric pneumonia diagnosis[J].Comput Methods Programs Biomed，2020，187：104964.
[62] IRFAN A，ADIVISHNU A L，SZE-TO A，et al.Classifying pneumonia among chest X-rays using transfer learning[C]//Annu Int Conf IEEE Eng Med Biol Soc，2020：2186-2189.
[63] APOSTOLOPOULOS I D，MPESIANA T A.Covid-19：automatic detection from X-ray images utilizing transfer learning with convolutional neural networks[J].Phys Eng Sci Med，2020，43（2）：635-640.
[64] GóMEZ-VALVERDE J J，ANTóN A，FATTI G，et al.Automatic glaucoma classification using color fundus images based on convolutional neural networks and transfer learning[J].Biomedical Optics Express，2019，10（2）：892-913.
[65] LIU T Y A，TING D S W，YI P H，et al.Deep learning and transfer learning for optic disc laterality detection：implications for machine learning in neuro-ophthalmology[J].J Neuroophthalmol，2020，40（2）：178-184.
[66] LE D，ALAM M，YAO C K，et al.Transfer learning for automated OCTA detection of diabetic retinopathy[J].Transl Vis Sci Technol，2020，9（2）：35.
[67] XUE L Y，JIANG Z Y，FU T T，et al.Transfer learning radiomics based on multimodal ultrasound imaging for staging liver fibrosis[J].Eur Radiol，2020，30（5）：2973-2983.
[68] LI H，HE L，DUDLEY J A，et al.DeepLiverNet：a deep transfer learning model for classifying liver stiffness using clinical and T2-weighted magnetic resonance imaging data in children and young adults[J].Pediatr Radiol，2021，51（3）：392-402.
[69] MAHBOD A，SCHAEFER G，WANG C，et al.Transfer learning using a multi-scale and multi-network ensemble for skin lesion classification[J].Comput Methods Programs Biomed，2020，193：105475.
[70] ZHUANG Z，KANG Y，JOSEPH RAJ A N，et al.Breast ultrasound lesion classification based on image decomposition and transfer learning[J].Med Phy，2020，47（12）：6257-6269.
[71] VU C C，SIDDIQUI Z A，ZAMDBORG L，et al.Deep convolutional neural networks for automatic segmentation of thoracic organs-at-risk in radiation oncology-use of non-domain transfer learning[J].J Appl Clin Med Phys，2020，21（6）：108-113.
[72] GHERARDINI M，MAZOMENOS E，MENCIASSI A，et al.Catheter segmentation in X-ray fluoroscopy using synthetic data and transfer learning with light U-nets[J].Computer Methods and Programs in Biomedicine，2020，192：105420.
[73] LO J，HEISLER M，VANZAN V，et al.Microvasculature segmentation and intercapillary area quantification of the deep vascular complex using transfer learning[J].Transl Vis Sci Technol，2020，9（2）：38.
[74] LOO J，FANG L，CUNEFARE D，et al.Deep longitudinal transfer learning-based automatic segmentation of photoreceptor ellipsoid zone defects on optical coherence tomography images of macular telangiectasia type 2[J].Biomedical Optics Express，2018，9（6）：2681-2698.
[75] JIANG Z，ZHANG H，WANG Y，et al.Retinal blood vessel segmentation using fully convolutional network with transfer learning[J].Comput Med Imaging Graph，2018，68：1-15.
[76] 赵丹，车南颖，宋志刚，等.基于深度迁移学习的肺癌病理诊断[J].中华病理学杂志，2020（11）：1120-1125.
ZHAO D，CHE N Y，SONG Z G，et al.Pathological diagnosis of lung cancer based on deep transfer learning[J].Chinese Journal of Pathology，2020（11）：1120-1125.
[77] ALEX V，VAIDHYA K，THIRUNAVUKKARASU S，et al.Semisupervised learning using denoising autoencoders for brain lesion detection and segmentation[J].J Med Imaging（Bellingham），2017，4（4）：041311.
[78] 马金林，魏萌马，马自萍.基于深度迁移学习的肺结节分割方法[J].计算机应用，2020，40（7）：2117-2125.
MA J L，WEI M M，MA Z P.Pulmonary nodule segmentation method based on deep transfer learning[J].Journal of Computer Application，2020，40（7）：2117-2125.
[79] SANFORD T H，ZHANG L，HARMON S A，et al.Data augmentation and transfer learning to improve generali-
zability of an automated prostate segmentation model[J].AJR Am J Roentgenol，2020，14：1-8.
[80] GRIMM F，EDL F，KERSCHER S R，et al.Semantic segmentation of cerebrospinal fluid and brain volume with a convolutional neural network in pediatric hydrocephalus-transfer learning from existing algorithms[J].Acta Neurochir（Wien），2020，162（10）：2463-2474.
[81] BYRA M，WU M，ZHANG X，et al.Knee menisci segmentation and relaxometry of 3D ultrashort echo time cones MR imaging using attention U-Net with transfer learning[J].Magn Reson Med，2020，83（3）：1109-1122.
[82] FLANNERY S W，KIAPOUR A M，EDGAR D J，et al.A transfer learning approach for automatic segmentation of the surgically treated anterior cruciate ligament[J].J Orthop Res，2021.
[83] JIANG H T，GUO J J，DU H W，et al.Transfer learning on T1-weighted images for brain age estimation[J].Math Biosci Eng，2019，16（5）：4382-4398.
[84] CHEN M，LI H，WANG J，et al.Early prediction of cognitive deficit in very preterm infants using brain structural connectome with transfer learning enhanced deep convolutional neural networks[J].Front Neurosci，2020，14：858.
[85] LOPEZ-GARCIA G，JEREZ J M，FRANCO L，et al.Transfer learning with convolutional neural networks for cancer survival prediction using gene-expression data[J].PLoS One，2020，15（3）：e0230536.
[86] HU L S，YOON H，ESCHBACHER J M，et al.Accurate patient-specific machine learning models of glioblastoma invasion using transfer learning[J].AJNR Am J Neuroradiol，2019，40（3）：418-425.
[87] SAMALA R K，CHAN H P，HADJIISKI L，et al.Mass detection in digital breast tomosynthesis：deep convolutional neural network with transfer learning from mammography[J].Med Phys，2016，43（12）：6654-6666.
[88] 唐思源，刘燕茹，杨敏.基于迁移学习和三维卷积神经网络检测肺结节[J].中国医学影像技术，2020，36（12）：1882-1886.
TANG S Y，LIU Y R，YANG M.Detection of pulmonary nodules based on transfer learning and three dimensional convolutional neural network[J].Chin J Med Imaging Technol，2020，36（12）：1882-1886.
[89] DUAN W，ZHANG J，ZHANG L，et al.Evaluation of an artificial intelligent hydrocephalus diagnosis model based on transfer learning[J].Medicine（Baltimore），2020，99（29）：e21229.
[90] SHAN H，ZHANG Y，YANG Q，et al.3D convolutional encoder-decoder network for low-dose CT via transfer learning from a 2D trained network[J].IEEE Trans Med Imaging，2018，37（6）：1522-1534.
[91] LIU H，WU J，LU W，et al.Noise reduction with cross-tracer and cross-protocol deep transfer learning for low-dose PET[J].Phys Med Biol，2020，65（18）：185006.
[92] HEGAZY M A A，CHO M H，LEE S Y.Image denoising by transfer learning of generative adversarial network for dental CT[J].Biomed Phys Eng Express，2020，6（5）：055024.
[93] 肖儿良，周莹，简献忠.迁移学习与GAN结合的医学图像融合模型[J].小型微型计算机系统，2020，41（9）：1972-1978.
XIAO E L，ZHOU Y，JIAN X Z.Medical image fusion model combining transfer learning and generative adversarial network[J].Journal of Chinese Computer Systems，2020，41（9）：1972-1978.
[94] GU Y，GE Z，BONNINGTON C P，et al.Progressive transfer learning and adversarial domain adaptation for cross-domain skin disease classification[J].IEEE Journal of Biomedical and Health Informatics，2020，24（5）：1379-1393.
[95] 黄盛，李菲菲，陈虬.基于改进深度残差网络的计算断层扫描图像分类算法[J].光学学报，2020，40（3）：56-64.
HUANG S，LI F F，CHEN Q.Computed tomography image classification algorithm based on improved deep residual network[J].Acta Optica Sinica，2020，40（3）：56-64.