Survey on Image Semantic Segmentation in Dilemma of Few-Shot

doi:10.3778/j.issn.1002-8331.2205-0496

Abstract

Abstract: In recent years, image semantic segmentation has developed rapidly due to the emergence of large-scale datasets. However, in practical applications, it is not easy to obtain large-scale, high-quality images, and image annotation also consumes a lot of manpower and time costs. In order to get rid of the dependence on the number of samples, few-shot semantic segmentation has gradually become a research hotspot. The current few-shot semantic segmentation methods mainly use the idea of meta-learning, which can be divided into three categories：based on the siamese neural network, based on the prototype network and based on the attention mechanism according to different model structures. Based on the current research, this paper introduces the development, advantages and disadvantages of various methods for few-shot semantic segmentation, as well as common datasets and experimental designs. On this basis, the application scenarios and future development directions are summarized.

Key words: semantic segmentation, meta-learning, few-shot learning

摘要： 近年来，由于大规模数据集的出现，图像语义分割技术得到快速发展。但在实际场景中，并不容易获取到大规模、高质量的图像，图像的标注也需要消耗大量的人力和时间成本。为了摆脱对样本数量的依赖，小样本语义分割技术逐渐成为研究热点。当前小样本语义分割的方法主要利用了元学习的思想，按照不同的模型结构可划分为基于孪生神经网络、基于原型网络和基于注意力机制三大类。基于近年来小样本语义分割的发展现状，介绍了小样本语义分割各类方法的发展及优缺点，以及小样本语义分割任务中常用的数据集及实验设计。在此基础上，总结了小样本语义分割技术的应用场景及未来的发展方向。

关键词: 语义分割, 元学习, 小样本学习

WEI Ting, LI Xinlei, LIU Hui. Survey on Image Semantic Segmentation in Dilemma of Few-Shot[J]. Computer Engineering and Applications, 2023, 59(2): 1-11.

韦婷, 李馨蕾, 刘慧. 小样本困境下的图像语义分割综述[J]. 计算机工程与应用, 2023, 59(2): 1-11.

References

[1] LONG J，SHELHAMER E，DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：3431-3440.
[2] CHEN L C，ZHU Y，PAPANDREOU G，et al.Encoder-decoder with atrous separable convolution for semantic imagesegmentation[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：801-818.
[3] LIU C，CHEN L C，SCHROFF F，et al.Auto-deeplab：hierarchical neural architecture search for semantic image segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2019：82-92.
[4] FU J，LIU J，TIAN H，et al.Dual attention network for scene segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2019：3146-3154.
[5] LATEEF F，RUICHEK Y.Survey on semantic segmentation using deep learning techniques[J].Neurocomputing，2019，338：321-348.
[6] THOMA M.A survey of semantic segmentation[J].arXiv：1602.06541，2016.
[7] VANSCHOREN J.Meta-learning：a survey[J].arXiv：1810.
03548，2018.
[8] FEI-FEI L，FERGUS R，PERONA P.One-shot learning of object categories[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2006，28（4）：594-611.
[9] KANG B，LIU Z，WANG X，et al.Few-shot object detection via feature reweighting[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：8420-8429.
[10] TIAN Y，WANG Y，KRISHNAN D，et al.Rethinking few-shot image classification：a good embedding is all you need?[C]//European Conference on Computer Vision.Cham：Springer，2020：266-282.
[11] SHABAN A，BANSAL S，LIU Z，et al.One-shot learning for semantic segmentation[C]//British Machine Vision Conference，2017：1-13.
[12] SNELL J，SWERSKY K，ZEMEL R.Prototypical networks for few-shot learning[C]//Advances in Neural Information Processing Systems，2017：4077-4087.
[13] XIE G S，XIONG H，LIU J，et al.Few-shot semantic segmentation with cyclic memory network[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：7273-7282.
[14] SCARSELLI F，GORI M，TSOI A C，et al.The graph neural network model[J].IEEE Transactions on Neural Networks，2008，20（1）：61-80.
[15] WANG Y，YAO Q，KWOK J T，et al.Generalizing from a few examples：a survey on few-shot learning[J].ACM Computing Surveys，2020，53（3）：1-34.
[16] 陈琼，杨咏，黄天林，等.小样本图像语义分割综述[J].数据与计算发展前沿，2021，3（6）：17-34.
CHEN Q，YANG Y，HUANG T L，et al.A survey on few-shot image semantic segmentation[J].Frontiers of Data and Computing，2021，3（6）：17-34.
[17] DONG N，XING E P.Few-shot semantic segmentation with prototype learning[C]//British Machine Vision Conference，2018.
[18] KULIS B.Metric learning：a survey[J].Foundations and Trendsin Machine Learning，2013，5（4）：287-364.
[19] 李凡长.元学习研究综述[J].计算机学报，2021（2）：422-446.
LI F C.A survey on recent advances in meta-learning[J].Chinese Journal of Computers，2021（2）：422-446.
[20] CHICCO D.Siamese neural networks：an overview[J].Artificial Neural Networks，2021：73-94.
[21] TIAN Z，ZHAO H，SHU M，et al.Prior guided feature enrichment network for few-shot segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2022，44（2）：1050-1065.
[22] ZHANG X，WEI Y，YANG Y，et al.Sg-one：similarity guidance network for one-shot semantic segmentation[J].IEEE Transactions on Cybernetics，2020，50（9）：3855-3865.
[23] YANG B，LIU C，LI B，et al.Prototype mixture models for few-shot semantic segmentation[C]//European Conference on Computer Vision.Cham：Springer，2020：763-778.
[24] WANG H，ZHANG X，HU Y，et al.Few-shot semantic segmentation with democratic attention networks[C]//European Conference on Computer Vision.Cham：Springer，2020：730-746.
[25] VASWANI A，SHAZEER N，PARMAR N，et al.Attention is all you need[C]//Advances in Neural Information Processing Systems，2017：5998-6008.
[26] BROMLEY J，GUYON I，LECUN Y，et al.Signature verification using a “siamese” time delay neural network[C]//Advances in Neural Information Processing Systems，1993.
[27] KOCH G，ZEMEL R，SALAKHUTDINOV R.Siamese neural networks for one-shot image recognition[C]//The International Conference on Machine Learning，2015.
[28] RAKELLY K，SHELHAMER E，DARRELL T，et al.Conditional networks for few-shot semantic segmentation[C]//International Conference on Learning Representations，2018：1-4.
[29] ZHANG C，LIN G，LIU F，et al.Canet：class-agnostic segmentation networks with iterative refinement and attentive few-shot learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：5212-5221.
[30] JIE L.Dynamic prototype convolution network for few-shot semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022.
[31] GAIROLA S，HEMANI M，CHOPRA A，et al.Simpropnet：improved similarity propagation for few-shot image segmentation[C]//Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence，2020：573-579.
[32] LANG C，CHENG G，TU B，et al.Learning what not to segment：a new perspective on few-shot segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2022.
[33] ZHAO H，SHI J，QI X，et al.Pyramid scene parsing network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：2881-2890.
[34] GATYS L A，ECKER A S，BETHGE M.A neural algorithm of artistic style[J].Journal of Vision，2016，16：326.
[35] SIAM M，ORESHKIN B N，JAGERSAND M.Amp：adaptive masked proxies for few-shot segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：5249-5258.
[36] NGUYEN K，TODOROVIC S.Feature weighting and boosting for few-shot segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：622-631.
[37] ZHANG B，XIAO J，QIN T.Self-guided and cross-guided learning for few-shot segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：8312-8321.
[38] WANG K，LIEW J H，ZOU Y，et al.Panet：few-shot image semantic segmentation with prototype alignment[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：9197-9206.
[39] LIU Y，ZHANG X，ZHANG S，et al.Part-aware prototype network for few-shot semantic segmentation[C]//European Conference on Computer Vision.Cham：Springer，2020：142-158.
[40] LI G，JAMPANI V，SEVILLA-LARA L，et al.Adaptive prototype learning and allocation for few-shot segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：8334-8343.
[41] MCLACHLAN G J，KRISHNAN T.The EM algorithm and extensions[M].[S.l.]：John Wiley & Sons，2007.
[42] ZHU K，ZHAI W，ZHA Z J，et al.Self-supervised tuning for few-shot segmentation[C]//Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence，2020：1019-1025.
[43] LIU B，DING Y，JIAO J，et al.Anti-aliasing semantic reconstruction for few-shot semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：9747-9756.
[44] 景庄伟，管海燕，彭代峰，等.基于深度神经网络的图像语义分割研究综述[J].计算机工程，2020，46（10）：1-17.
JING Z W，GUAN H Y，PENG D F，et al.Survey of research in image semantic segmentation based on deep neural network[J].Computer Engineering，2020，46（10）：1-17.
[45] GUO H，ZHENG K，FAN X，et al.Visual attention consistency under image transforms for multi-label image classification[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：729-739.
[46] PANG Y，XIE J，KHAN M H，et al.Mask-guided attention network for occluded pedestrian detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：4967-4975.
[47] 欧阳柳，贺禧，瞿绍军.全卷积注意力机制神经网络的图像语义分割[J].计算机科学与探索，2022，16（5）：1136-1145.
OUYANG L，HE X，QU S J.Fully convolutional neural network with attention module for semantic segmentation[J].Journal of Frontiers of Computer Science and Technology，2022，16（5）：1136-1145.
[48] YANG Y，MENG F，LI H，et al.A new local transformation module for few-shot segmentation[C]//International Conference on Multimedia Modeling.Berlin：Springer，2020：76-87.
[49] 刘宇轩，孟凡满，李宏亮，等.一种结合全局和局部相似性的小样本分割方法[J].北京航空航天大学学报，2021，47（3）：665-674.
LIU Y X，MENG F M，LI H L，et al.A few shot segmentation method combining global and local similarity[J].Journal of Beijing University of Aeronautics and Astronautics，2021，47（3）：665-674.
[50] 徐辉，祝玉华，甄彤，等.深度神经网络图像语义分割方法综述[J].计算机科学与探索，2021，15（1）：47-59.
XU H，ZHU Y H，ZHEN T，et al.Survey of image semantic segmentation methods based on deep neural network[J].Journal of Frontiers of Computer Science and Technology，2021，15（1）：47-59.
[51] YANG X，WANG B，CHEN K，et al.Brinet：towards bridging the intra-class and inter-class gaps in one-shot segmentation[C]//British Machine Vision Conference，2020.
[52] LU Z，HE S，ZHU X，et al.Simpler is better：few-shot semantic segmentation with classifier weight transformer[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：8741-8750.
[53] MIN J，KANG D，CHO M.Hypercorrelation squeeze for few-shot segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：6941-6952.
[54] LI X，WEI T，CHEN Y P，et al.Fss-1000：a 1000-class dataset for few-shot segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：2869-2878.
[55] CODELLA N C F，GUTMAN D，CELEBI M E，et al.Skin lesion analysis toward melanoma detection：a challenge at the 2017 International Symposium on Biomedical Imaging，hosted by the international skin imaging collaboration[C]//2018 IEEE 15th International Symposium on Biomedical Imaging，2018：168-172.
[56] MENDON?A T，FERREIRA P M，MARQUES J S，et al.PH2-a dermoscopic image database for research and benchmarking[C]//2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society，2013：5437-5440.
[57] LANDMAN B，XU Z B，IGELSIAS J，et al.Miccaimulti-atlas labeling beyond the cranial vault-workshop and challenges[C]//MICCAIMulti-Atlas Labeling Beyond the Cranial Vault-Workshop and Challenges，2015.
[58] KAVUR A E，GEZER N S，BARI? M，et al.CHAOS challenge-combined（CT-MR） healthy abdominal organ segmentation[J].Medical Image Analysis，2021，69：101950.
[59] EVERINGHAM M，ESLAMI S M，VAN GOOL L，et al.The pascal visual object classes challenge：a retrospective[J].International Journal of Computer Vision，2015，111（1）：98-136.
[60] HARIHARAN B，ARBELáEZ P，BOURDEV L，et al.Semantic contours from inverse detectors[C]//2011 International Conference on Computer Vision，2011：991-998.
[61] LIN T Y，MAIRE M，BELONGIE S，et al.Microsoft coco：common objects in context[C]//European Conference on Computer Vision.Cham：Springer，2014：740-755.
[62] RUSSAKOVSKY O，DENG J，SU H，et al.Imagenet large scale visual recognition challenge[J].International Journal of Computer Vision，2015，115（3）：211-252.
[63] REZATOFIGHI H，TSOI N，GWAK J Y，et al.Generalized intersection over union：a metric and a loss for bounding box regression[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：658-666.
[64] CHEN L C，PAPANDREOU G，KOKKINOS I，et al.Deeplab：semantic image segmentation with deep convolutional nets，atrous convolution，and fully connected CRFs[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，40（4）：834-848.
[65] TANG H，LIU X，SUN S，et al.Recurrent mask refinement for few-shot medical image segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2021：3918-3928.
[66] OUYANG C，BIFFI C，CHEN C，et al.Self-supervision with superpixels：training few-shot medical image segmentation without annotation[C]//European Conference on Computer Vision.Cham：Springer，2020：762-780.
[67] FEYJIE A R，AZAD R，PEDERSOLI M，et al.Semi-supervised few-shot learning for medical image segmentation[J].arXiv：2003.08462，2020.
[68] GUO Y，WANG H，HU Q，et al.Deep learning for 3d point clouds：a survey[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2020，43（12）：4338-4364.
[69] CHEN X，ZHANG C，LIN G，et al.Compositional prototype network with multi-view comparision for few-shot point cloud semantic segmentation[J].arXiv：2012.14255，2020.
[70] 王俊杰，张军航.基于多尺度特征融合的小样本遥感图像分割[J].华中科技大学学报（自然科学版），2022，50（3）：62-67.
WANG J J，ZHANG J H.Small sample remote sensing image segmentation based on multiscale feature fusion[J].Journal of Huazhong University of Science and Technology（Natural Science Edition），2022，50（3）：62-67.
[71] 朱祯悦，吕淑静，吕岳.基于图匹配网络的小样本违禁物品分割算法[J].红外与激光工程，2021，50（11）：418-426.
ZHU Z Y，LV S J，LV Y.Few-shot prohibited item segmentation algorithm based on graph matching network[J].Infrared and Laser Engineering，2021，50（11）：418-426.
[72] 许国良，毛骄.基于协同注意力的小样本的手机屏幕缺陷分割[J].电子与信息学报，2022，44（4）：1476-1483.
XU G L，MAO J.Few-shot segmentation on mobile phone screen defect based on co-attention[J].Journal of Electronics & Information Technology，2022，44（4）：1476-1483.
[73] 刘颖，雷研博，范九伦，等.基于小样本学习的图像分类技术综述[J].自动化学报，2021，47（2）：297-315.
LIU Y，LEI Y B，FAN J L，et al.Survey on image classification technology based on small sample learning[J].Acta Automatica Sinica，2021，47（2）：297-315.
[74] 张振伟，郝建国，黄健，等.小样本图像目标检测研究综述[J].计算机工程与应用，2022，58（5）：1-11.
ZHANG Z W，HAO J G，HUANG J，et al.Review of few-shot object detection[J].Computer Engineering and Applications，2022，58（5）：1-11.
[75] ZHOU J，CUI G，HU S，et al.Graph neural networks：a review of methods and applications[J].AI Open，2020，1：57-81.
[76] 徐冰冰，岑科廷，黄俊杰，等.图卷积神经网络综述[J].计算机学报，2020，43（5）：755-780.
XU B B，CEN K T，HUANG J J，et al.A survey on graph convolutional neural network[J].Chinese Journal of Computers，2020，43（5）：755-780.
[77] XIE G S，LIU J，XIONG H，et al.Scale-aware graph neural network for few-shot semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：5475-5484.
[78] QI X，LIAO R，JIA J，et al.3D graph neural networks for RGBD semantic segmentation[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：5199-5208.
[79] ZHANG C，LIN G，LIU F，et al.Pyramid graph networks with connection attentions for region-based one-shot semantic segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：9587-9595.

[80] 贾熹滨，李佳.金字塔原型对齐的轻量级小样本语义分割网络[J].北京工业大学学报，2021，47（5）：455-462.

JIA X B，LI J.Lightweight pyramid prototype alignment network for few- shot semantic segmentation[J].Journal of Beijing University of Technology，2021，47（5）：455-462.