Review of Application of Deep Learning in Colon Polyp Segmentation

doi:10.3778/j.issn.1002-8331.2303-0124

Abstract

Abstract: Most colorectal cancers originate from malignant lesions of colon polyps. It is of great clinical significance to use computer-aided diagnosis system to automatically and accurately segment colon polyps, which can help doctors improving the detection rate of polyps during colonoscopy. Nowadays, deep learning technology is widely used in medical image segmentation, and the colon polyp segmentation algorithm based on deep learning has also made significant progress. Firstly, the traditional polyp segmentation algorithm and its advantages and limitations are briefly introduced. Secondly, the deep learning polyp segmentation algorithm is reviewed in three aspects：segmentation model based on classical CNN structure, U-Net structure, and multi-model fusion, then the improvement strategy of the algorithm and its advantages and limitations is summarized. The public datasets of the colon polyp image and the data preprocessing methods are concluded. Finally, the challenges of polyp segmentation based on deep learning are summarized, and the future research direction in this field is prospected.

Key words: colon polyps, deep learning, medical imaging segmentation, polyp segmentation

摘要： 大部分结直肠癌起源于结肠息肉的恶性病变，使用计算机辅助诊断系统实现结肠息肉的自动精准分割具有重要的临床意义，能够在结肠镜检查过程中辅助医生提高息肉检出率。目前深度学习技术在医学图像分割领域应用广泛，基于深度学习的结肠息肉分割算法也取得了重大进展。简要介绍了传统息肉分割算法及其优点和局限性。重点从三个方面对深度学习息肉分割算法进行综述：基于经典CNN结构、基于U-Net结构和基于多模型融合的分割模型，并总结算法改进策略及其优势和局限性。归纳结肠息肉图像公开数据集及数据预处理方法，最后总结基于深度学习的息肉分割研究面临的挑战，并对该领域未来的研究方向做出展望。

关键词: 结肠息肉, 深度学习, 医学图像分割, 息肉分割

SUN Fuyan, WANG Qiong, LYU Zongwang, GONG Chunyan. Review of Application of Deep Learning in Colon Polyp Segmentation[J]. Computer Engineering and Applications, 2023, 59(23): 15-27.

孙福艳, 王琼, 吕宗旺, 龚春艳. 深度学习在结肠息肉分割中的应用综述[J]. 计算机工程与应用, 2023, 59(23): 15-27.

References

[1] 王凯悦，蔡善荣，张苏展.不同筛查手段对结直肠腺瘤检出的研究进展[J].实用肿瘤杂志，2022，37（3）：283-291.
WANG K Y，CAI S R，ZHANG S Z.Research progress on different screening methods for detecting colorectal adenomas[J].Journal of Practical Oncology，2022，37（3）：283-291.
[2] SANCHEZ-PERALTA L F，BOTE-CURIEL L，PICON A，et al.Deep learning to find colorectal polyps in colonoscopy：a systematic literature review[J].Artificial Intelligence in Medicine，2020，108：101923.
[3] NOGUEIRA-RODRíGUEZ A，DOMíNGUEZ-CARBAJALES R，LóPEZ-FERNáNDEZ H，et al.Deep neural networks approaches for detecting and classifying colorectal polyps[J].Neurocomputing，2021，423：721-734.
[4] QUAN S Y，WEI M T，LEE J，et al.Clinical evaluation of a real-time artificial intelligence-based polyp detection system：a US multi-center pilot study[J].Scientific Reports，2022，12（1）：6598.
[5] HSU C M，HSU C C，HSU Z M，et al.Colorectal polyp image detection and classification through grayscale images and deep learning[J].Sensors，2021，21（18）：5995.
[6] XIA S，KRISHNAN S M，TJOA M P，et al.A novel methodology for extracting colon’s lumen from colonoscopic images[J].Journal of Systemics，Cybernetics and Informatics，2003，1（2）：7-12.
[7] WANG Z，LI L，ANDERSON J，et al.Computer-aided detection and diagnosis of colon polyps with morphological and texture features[C]//Medical Imaging 2004：Image Processing，2004：972-979.
[8] TAJBAKHSH N，GURUDU S R，LIANG J.Automatic polyp detection using global geometric constraints and local intensity variation patterns[C]//Proceedings of the 17th International Conference on Medical Image Computing and Computer-Assisted Intervention，Boston，MA，USA，September 14-18，2014：179-187.
[9] HWANG S，OH J H，TAVANAPONG W，et al.Polyp detection in colonoscopy video using elliptical shape feature[C]//Proceedings of the 2007 IEEE International Conference on Image Processing，2007：465-468.
[10] CHOWDHURY T A，GHITA O，WHELAN P F.A statistical approach for robust polyp detection in CT colonography[C]//Proceedings of the 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference，2006：2523-2526.
[11] LI P，CHAN K L，KRISHNAN S M.Learning a multi-size patch-based hybrid kernel machine ensemble for abnormal region detection in colonoscopic images[C]//Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition（CVPR’05），2005：670-675.
[12] IAKOVIDIS D K，KOULAOUZIDIS A.Automatic lesion detection in wireless capsule endoscopy—a simple solution for a complex problem[C]//Proceedings of the 2014 IEEE International Conference on Image Processing（ICIP），2014：2236-2240.
[13] MAGHSOUDI O H.Superpixel based segmentation and classification of polyps in wireless capsule endoscopy[C]//2017 IEEE Signal Processing in Medicine and Biology Symposium（SPMB），2017：1-4.
[14] LECUN Y，BOSER B，DENKER J S，et al.Backpropagation applied to handwritten zip code recognition[J].Neural Computation，1989，1（4）：541-551.
[15] LONG J，SHELHAMER E，DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：3431-3440.
[16] QADIR H A，SHIN Y，SOLHUSVIK J，et al.Polyp detection and segmentation using mask R-CNN：does a deeper feature extractor CNN always perform better?[C]//Proceedings of the 13th International Symposium on Medical Information and Communication Technology（ISMICT），2019：1-6.
[17] AKBARI M，MOHREKESH M，NASR-ESFAHANI E，et al.Polyp segmentation in colonoscopy images using fully convolutional network[C]//Proceedings of the 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society（EMBC），Honolulu，HI，USA，2018：69-72.
[18] ZHANG L，DOLWANI S，YE X.Automated polyp segmentation in colonoscopy frames using fully convolutional neural network and textons[C]//Proceedings of the 21st Annual Conference on Medical Image Understanding and Analysis，Edinburgh，UK，July 11-13，2017：707-717.
[19] BRANDAO P，MAZOMENOS E，CIUTI G，et al.Fully convolutional neural networks for polyp segmentation in colonoscopy[C]//Medical Imaging 2017：Computer-Aided Diagnosis，2017：101-107.
[20] BRANDAO P，ZISIMOPOULOS O，MAZOMENOS E，et al.Towards a computed-aided diagnosis system in colonoscopy：automatic polyp segmentation using convolution neural networks[J].Journal of Medical Robotics Research，2018，3（2）：1840002.
[21] JORDAN M I.Serial order：a parallel distributed processing approach[J] North-Holland，1997，121：471-495.
[22] HOCHREITER S，SCHMIDHUBER J.Long short-term memory[J] Neural Computation，1997，9（8）：1735-1780.
[23] XIAO W T，CHANG L J，LIU W M.Semantic segmentation of colorectal polyps with DeepLab and LSTM networks[C]//Proceedings of the 2018 IEEE International Conference on Consumer Electronics-Taiwan（ICCE-TW），2018：1-2.
[24] GOODFELLOW I，POUGET-ABADIE J，MIRZA M，et al.Generative adversarial nets in advances in neural information processing systems（NIPS）[M].NY，USA：Curran Associates，Inc Red Hook，2014：2672-2680.
[25] XIE X，CHEN J，LI Y，et al.MI2 GAN：generative adversarial network for medical image domain adaptation using mutual information constraint[C]//Proceedings of the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention，Lima，Peru，October 4-8，2020：516-525.
[26] CHEN J，LI Y，MA K，et al.Generative adversarial networks for video-to-video domain adaptation[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2020：3462-3469.
[27] AHMED A，ALI M A.Generative adversarial networks for automatic polyp segmentation[J].arXiv：2012.06771，2020.
[28] GUO X，YANG C，LIU Y，et al.Learn to threshold：Thresholdnet with confidence-guided manifold mixup for polyp segmentation[J].IEEE Transactions on Medical Imaging，2020，40（4）：1134-1146.
[29] THANH N C，LONG T Q.CRF-EfficientUNet：an improved UNet framework for polyp segmentation in colonoscopy images with combined asymmetric loss function and CRF-RNN layer[J].IEEE Access，2021，9：156987-157001.
[30] ?ZTüRK ?，?ZKAYA U.Gastrointestinal tract classification using improved LSTM based CNN[J].Multimedia Tools and Applications，2020，79（39/40）：28825-28840.
[31] ITOH H，ROTH H R，LU L，et al.Towards automated colonoscopy diagnosis：binary polyp size estimation via unsupervised depth learning[C]//Proceedings of the 21st International Conference on Medical Image Computing and Computer Assisted Intervention，Granada，Spain，September 16-20，2018：611-619.
[32] MOREU E，MCGUINNESS K，O’CONNOR N E.Synthetic data for unsupervised polyp segmentation[J].arXiv：2202.08680，2022.
[33] RONNEBERGER O，FISCHER P，BROX T.U-Net：convolutional networks for biomedical image segmentation[C]//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention，Munich，Germany，October 5-9，2015：234-241.
[34] MAHMUD T，PAUL B，FATTAH S A.PolypSegNet：a modified encoder-decoder architecture for automated polyp segmentation from colonoscopy images[J].Computers in Biology and Medicine，2021，128：104119.
[35] JHA D，RIEGLER M A，JOHANSEN D，et al.Doubleu-Net：a deep convolutional neural network for medical image segmentation[C]//Proceedings of the IEEE 33rd International Symposium on Computer-Based Medical Systems（CBMS），Rochester，MN，USA，2020：558-564.
[36] 刘佳伟，刘巧红，李晓欧，等.一种改进的双U型网络的结肠息肉分割方法[J].光学学报，2021，41（18）：72-80.
LIU J W，LIU Q H，LI X O，et al.Improved colonic polyp segmentation method based on double U-shaped network[J].Acta Optica Sinica，2021，41（18）：72-80.
[37] WEI J，HU Y，ZHANG R，et al.Shallow attention network for polyp segmentation[C]//Proceedings of the 24th International Conference on Medical Image Computing and Computer Assisted Intervention，Strasbourg，France，September 27-October 1，2021：699-708.
[38] KIM T，LEE H，KIM D.UACANet：uncertainty augmented context attention for polyp segmentation[C]//Proceedings of the 29th ACM International Conference on Multimedia，2021：2167-2175.
[39] TOMAR N K，JHA D，BAGCI U，et al.TGANet：text-guided attention for improved polyp segmentation[C]// Proceedings of the 25th International Conference on Medical Image Computing and Computer Assisted Intervention，Singapore，September 18-22，2022：151-160.
[40] DOSOVITSKIY A，BEYER L，KOLESNIKOV A，et al.An image is worth 16×16 words：transformers for image recognition at scale[J].arXiv：2010.11929，2020.
[41] CHEN J，LU Y，YU Q，et al.Transunet：transformers make strong encoders for medical image segmentation[J].arXiv：2102.04306，2021.
[42] ZHANG Y，LIU H，HU Q.Transfuse：fusing transformers and cnns for medical image segmentation[C]//Proceedings of the 24th International Conference on Medical Image Computing and Computer Assisted Intervention，Strasbourg，France，September 27-October 1，2021：14-24.
[43] WU H，ZHONG J，WANG W，et al.Precise yet efficient semantic calibration and refinement in convnets for real-time polyp segmentation from colonoscopy videos[C]//Proceedings of the AAAI Conference on Artificial Intelligence，2021：2916-2924.
[44] PATEL K，BUR A M，WANG G.Enhanced U-Net：a feature enhancement network for polyp segmentation[C]// Proceedings of the 18th Conference on Robots and Vision（CRV），2021：181-188.
[45] ZHAO X，ZHANG L，LU H.Automatic polyp segmentation via multi-scale subtraction network[C]//Proceedings of the 24th International Conference on Medical Image Computing and Computer Assisted Intervention，Strasbourg，France，September 27-October 1，2021：120-130.
[46] TOMAR N K，SRIVASTAVA A，BAGCI U，et al.Automatic polyp segmentation with multiple kernel dilated convolution network[C]//Proceedings of the IEEE 35th International Symposium on Computer-Based Medical Systems（CBMS），2022：317-322.
[47] 时永刚，李祎，周治国，等.基于阶梯结构的U-Net结肠息肉分割算法[J].电子与信息学报，2022，44（1）：39-47.
SHI Y G，LI Y，ZHOU Z G，et al.Polyp segmentation using stair-structured U-Net[J].Journal of Electronics & Information Technology，2022，44（1）：39-47.
[48] JI G P，XIAO G，CHOU Y C，et al.Video polyp segmentation：a deep learning perspective[J].Machine Intelligence Research，2022：1-19.
[49] FAN D P，JI G P，ZHOU T，et al.PraNet：parallel reverse attention network for polyp segmentation[C]//Proceedings of the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention，Lima，Peru，October 4-8，2020：263-273.
[50] LIN A，CHEN B，XU J，et al.DS-Transunet：dual swin transformer u-net for medical image segmentation[J].IEEE Transactions on Instrumentation and Measurement，2022，71：1-15.
[51] DONG B，WANG W，FAN D P，et al.Polyp-PVT：polyp segmentation with pyramid vision transformers[J].arXiv：2108.06932，2021.
[52] ZHANG R，LI G，LI Z，et al.Adaptive context selection for polyp segmentation[C]//Proceedings of the 23rd International Conference on Medical Image Computing and Computer Assisted Intervention，Lima，Peru，October 4-8，2020：253-262.
[53] TOMAR N K.Automatic polyp segmentation using fully convolutional neural network[J].arXiv：2101.04001，2021.
[54] JHA D，SMEDSRUD P H，JOHANSEN D，et al.A comprehensive study on colorectal polyp segmentation with ResUNet++，conditional random field and test-time augmentation[J].IEEE Journal of Biomedical and Health Informatics，2021，25（6）：2029-2040.
[55] BANIK D，ROY K，BHATTACHARJEE D，et al.Polyp-Net：a multimodel fusion network for polyp segmentation[J].IEEE Transactions on Instrumentation and Measurement，2020，70：1-12.
[56] CHEN H，QI X，YU L，et al.DCAN：deep contour-aware networks for accurate gland segmentation[C]//Proceedings of the IEEE conference on Computer Vision and Pattern Recognition，2016：2487-2496.
[57] TAN C，ZHAO L，YAN Z，et al.Deep multi-task and task-specific feature learning network for robust shape preserved organ segmentation[C]//Proceedings of the IEEE 15th International Symposium on Biomedical Imaging（ISBI 2018），2018：1221-1224.
[58] SHEN H，WANG R，ZHANG J，et al.Boundary-aware fully convolutional network for brain tumor segmentation[C]//Proceedings of the 20th International Conference on Medical Image Computing and Computer-Assisted Intervention，Quebec City，QC，Canada，September 11-13，2017：433-441.
[59] FANG Y，CHEN C，YUAN Y，et al.Selective feature aggregation network with area-boundary constraints for polyp segmentation[C]//Proceedings of the 22nd International Conference on Medical Image Computing and Computer Assisted Intervention（MICCAI 2019），Shenzhen，China，October 13-17，2019：302-310.
[60] FANG Y，ZHU D，YAO J，et al.ABC-Net：area-boundary constraint network with dynamical feature selection for colorectal polyp segmentation[J].IEEE Sensors Journal，2020，21（10）：11799-11809.
[61] MURUGESAN B，SARVESWARAN K，SHANKARANARAYANA S M，et al.Psi-Net：shape and boundary aware joint multi-task deep network for medical image segmentation[C]//2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society（EMBC），2019：7223-7226.
[62] BERNAL J，TAJKBAKSH N，SANCHEZ F J，et al.Comparative validation of polyp detection methods in video colonoscopy：results from the MICCAI 2015 endoscopic vision challenge[J].IEEE Transactions on Medical Imaging，2017，36（6）：1231-1249.
[63] POGORELOV K，RANDEL K R，GRIWODZ C，et al.Kvasir：a multi-class image dataset for computer aided gastrointestinal disease detection[C]//Proceedings of the 8th ACM on Multimedia Systems Conference，2017：164-169.
[64] MA Y，CHEN X，CHENG K，et al.LDPolypVideo benchmark：a large-scale colonoscopy video dataset of diverse polyps[C]//Proceedings of the 24th International Conference on Medical Image Computing and Computer Assisted Intervention，Strasbourg，France，September 27-October 1，2021：387-396.
[65] MESEJO P，PIZARRO D，ABERGEL A，et al.Computer-aided classification of gastrointestinal lesions in regular colonoscopy[J].IEEE Transactions on Medical Imaging，2016，35（9）：2051-2063.
[66] SILVA J，HISTACE A，ROMAIN O，et al.Toward embedded detection of polyps in wce images for early diagnosis of colorectal cancer[J].International Journal of Computer Assisted Radiology and Surgery，2014，9：283-293.
[67] BERNAL J，SáNCHEZ F J，FERNáNDEZ-ESPARRACH G，et al.WM-DOVA maps for accurate polyp highlighting in colonoscopy：validation vs.saliency maps from physicians[J].Computerized Medical Imaging and Graphics，2015，43：99-111.
[68] BERNAL J，SáNCHEZ J，VILARINO F.Towards automatic polyp detection with a polyp appearance model[J].Pattern Recognition，2012，45（9）：3166-3182.
[69] VáZQUEZ D，BERNAL J，SáNCHEZ F J，et al.A benchmark for endoluminal scene segmentation of colonoscopy images[J].Journal of Healthcare Engineering，2017.
[70] ZHANG X，CHEN F，YU T，et al.Real-time gastric polyp detection using convolutional neural networks[J].PloS One，2019，14（3）：e0214133.
[71] JHA D，SMEDSRUD P H，RIEGLER M A，et al.Kvasir-seg：a segmented polyp dataset[C]//Proceedings of the 26th International Conference on MultiMedia Modeling，Daejeon，South Korea，January 5-8，2020：451-462.
[72] TAJBAKHSH N，GURUDU S R，LIANG J.Automated polyp detection in colonoscopy videos using shape and context information[J].IEEE Transactions on Medical Imaging，2015，35（2）：630-644.
[73] ANGERMANN Q，BERNAL J，SáNCHEZ-MONTES C，et al.Towards real-time polyp detection in colonoscopy videos：adapting still frame-based methodologies for video sequences analysis[C]//Proceedings of the 4th International Workshop on Computer Assisted and Robotic Endoscopy and Clinical Image-Based Procedures，and 6th International Workshop on Held in Conjunction with MICCAI 2017，Québec City，QC，Canada，September 14，2017：29-41.
[74] BERNAL J J，HISTACE A，MASANA M，et al.Polyp detection benchmark in colonoscopy videos using gtcreator：a novel fully configurable tool for easy and fast annotation of image databases[C]//Proceedings of the 32nd CARS Conference，2018.
[75] MISAWA M，KUDO S，MORI Y，et al.Development of a computer-aided detection system for colonoscopy and a publicly accessible large colonoscopy video database（with video）[J].Gastrointestinal Endoscopy，2021，93（4）：960-967.
[76] ITOH H，MISAWA M，MORI Y，et al.SUN colonoscopy video database[EB/OL].（2020）[2023-02-01].http：//amed8k.sundatabase.org/
[77] 王亚刚，郗怡媛，潘晓英.改进DeepLabv3+网络的肠道息肉分割方法[J].计算机科学与探索，2020，14（7）：1243-1250.
WANG Y G，XI Y Y，PAN X Y.Method for intestinal polyp segmentation by improving DeepLabv3+ network[J].Journal of Frontiers of Computer Science and Technology，2020，14（7）：1243-1250.
[78] ZHANG Q，JIANG Z，LU Q，et al.Split to be slim：an overlooked redundancy in vanilla convolution[J].arXiv：2006.12085，2020.