Efficient Cross-Domain Transformer Few-Shot Semantic Segmentation Network

doi:10.3778/j.issn.1002-8331.2209-0156

Abstract

Abstract: Few-shot semantic segmentation aims at only using several labeling samples to learn target features and complete the semantic segmentation task. The main problems in mainstream research are low training efficiency, meta training and meta testing in the same data domain. For this task, this paper proposes an efficient, cross-domain few-shot semantic segmentation network based on Transformer: SGFNet. In the encoding layer, use the shared weight MixVisionTransformer to build a siamese network to extract the support set and query set image features. In the relationship calculation layer, calculate the Hadamard product of the support set image feature vector and its corresponding mask to extract the target feature maps, and calculate the relationship between them and the image features of the query set. In the decoder layer, improve the MLP decoder and propose a residual decoder to decode the features of different hierarchies to obtain the final segmentation result. Experiments show that the model only needs to use a single 3090 GPU on the FSS-1000 dataset for training 1.5~4.0 h to get the optimal result 1-shot mIoU 87.0% on PASCAL-5i and the COCO-20i dataset perform cross-domain tests to achieve non-cross-domain effects, the 1-shot mIoU is 60.4% and 33.0%, respectively, proving that the model is efficient and cross-domain.

Key words: few-shot semantic segmentation (FSS), cross-domain, transformer, few-shot learning (FSL), semantic segmentation

摘要： 小样本语义分割旨在仅使用数个标注样本学习目标类别特征并完成分割任务。主流研究存在的主要问题是：训练效率低下，训练和测试在同一数据域。为此构建了一种基于Transformer的高效、跨域的小样本语义分割网络SGFNet。在编码层，使用共享权重的MixVisionTransformer构建孪生网络，用于提取支持集和查询集的图像特征；在关系计算层，通过计算支持集图像特征向量与其对应mask的哈达玛积，提取目标类别的高维特征，并与查询集图像特征进行关系计算；在解码层，改进基于MLP的解码器，提出了残差解码器，将不同层级的特征解码得到最终分割结果。实验表明，该模型只需要在FSS-1000数据集上使用单张3090 GPU训练1.5~4.0 h，即可在FSS-1000数据集上获得最优结果1-shot mIoU 87.0%，在PASCAL-5i和COCO-20i数据集进行跨域测试达到非跨域的效果，1-shot mIoU分别为60.4%和33.0%，证明了该模型高效且跨域。

关键词: 小样本语义分割（FSS）, 跨域, Transformer, 小样本学习（FSL）, 语义分割

FANG Hong, LI Desheng, JIANG Guangjie. Efficient Cross-Domain Transformer Few-Shot Semantic Segmentation Network[J]. Computer Engineering and Applications, 2024, 60(4): 142-152.

方红, 李德生, 蒋广杰. 高效跨域的Transformer小样本语义分割网络[J]. 计算机工程与应用, 2024, 60(4): 142-152.

References

[1] XING E, JORDAN M, RUSSELL S J, et al. Distance metric learning with application to clustering with side-information[C]//Advances in Neural Information Processing Systems 15, 2002: 505-512.
[2] 陈琼, 杨咏, 黄天林, 等. 小样本图像语义分割综述[J]. 数据与计算发展前沿, 2021, 3(6): 17-34.
CHEN Q, YANG Y, HUANG T L, et al. A survey on few-shot image semantic segmentation[J]. Frontiers of Data & Computing. 2021, 3(6): 17-34.
[3] ZHANG C, LIN G, LIU F, et al. CANet: class-agnostic segmentation networks with iterative refinement and attentive few-shot learning[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, Jun 16-20, 2019. Piscataway: IEEE, 2019: 5217-5226.
[4] TIAN Z, ZHAO H, SHU M, et al. Prior guided feature enrichment network for few-shot segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(2): 1050-1065.
[5] XIE G, LIU J, XIONG H, et al. Scale-aware graph neural network for few-shot semantic segmentation[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, Jun 19-25, 2021. Piscataway: IEEE, 2021: 5475-5484.
[6] ZHANG C, LIN G, LIU F, et al. Pyramid graph networks with connection attentions for region-based one-shot semantic segmentation[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Long Beach,Jun 16-20, 2019. Piscataway: IEEE, 2019: 9587-9595.
[7] SCARSELLI F, GORI M, TSOI A C, et al. The graph neural network model[J]. IEEE Transactions on Neural Networks, 2008, 20(1): 61-80.
[8] LU Z, HE S, ZHU X, et al. Simpler is better: few-shot semantic segmentation with classifier weight transformer[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision,?Montreal, Oct 11-17, 2021. Piscataway: IEEE, 2021: 8741-8750.
[9] MIN J, KANG D, CHO M. Hypercorrelation squeeze for few-shot segmentation[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision,?Montreal, Oct 11-17, 2021. Piscataway: IEEE, 2021: 6941-6952.
[10] SNELL J, SWERSKY K, ZEMEL R. Prototypical networks for few-shot learning[C]//Advances in Neural Information Processing Systems 30, 2017: 4077-4087.
[11] WANG K, LIEW J H, ZOU Y, et al. PANet: few-shot image semantic segmentation with prototype alignment[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov2, 2019. Piscataway: IEEE, 2019: 9197-9206.
[12] LIU W, ZHANG C, LIN G, et al. CRNet: cross-reference networks for few-shot segmentation[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 14-19, 2020. Piscataway: IEEE, 2020: 4165-4173.
[13] BOUDIAF M, KERVADEC H, MASUD Z I, et al. Few-shot segmentation without meta-learning: a good transductive inference is all you need?[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition,?Nashville, Jun 19-25, 2021. Piscataway:IEEE, 2021: 13979-13988.
[14] LI G, JAMPANI V, SEVILLA-LARA L, et al. Adaptive prototype learning and allocation for few-shot segmentation[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, Jun 19-25, 2021. Piscataway: IEEE, 2021: 8334-8343.
[15] XIE E, WANG W, YU Z, et al. SegFormer: simple and efficient design for semantic segmentation with transformers[C]//Advances in Neural Information Processing Systems 34, 2021: 12077-12090.
[16] SANTORO A, BARTUNOV S, BOTVINICK M, et al. Meta-learning with memory-augmented neural networks[C]//Proceedings of the 33rd International Conference on Machine Learning, New York, Jun 19-24, 2016: 1842-1850.
[17] KOCH G, ZEMEL R, SALAKHUTDINOV R. Siamese neural networks for one-shot image recognition[C]//Proceedings of the 32nd International Conference on Machine Learning, Lille, 2015.
[18] VINYALS O, BLUNDELL C, LILLICRAP T, et al. Matching networks for one shot learning[C]//Advances in Neural Information Processing Systems 29, 2016: 3630-3638.
[19] YANG B, LIU C, LI B, et al. Prototype mixture models for few-shot semantic segmentation[C]//Proceedings of the 16th European Conference on Computer Vision, Glasgow, Aug 23-28, 2020. Cham: Springer, 2020: 763-778.
[20] FINN C, ABBEEL P, LEVINE S. Model-agnostic meta-learning for fast adaptation of deep networks[C]//Proceedings of the 34th International Conference on Machine Learning, Sydney, Aug 6-11, 2017: 1126-1135.
[21] 张鑫, 姚庆安, 赵健, 等. 全卷积神经网络图像语义分割方法综述[J]. 计算机工程与应用, 2022, 58(8): 45-57.
ZHANG X, YAO Q A, ZHAO J, et al. Image semantic segmentation based on fully convolutional neural network[J]. Computer Engineering and Applications, 2022, 58(8): 45-57.
[22] DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: Transformers for image recognition at scale[J]. arXiv:2010.11929, 2020.
[23] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]//Proceedings of the 2015 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Boston, Jun 8-10, 2015. Piscataway: IEEE, 2015: 3431-3440.
[24] RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]// Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Oct 5-9, 2015. Berlin: Springer, 2015: 234-241.
[25] BADRINARAYANAN V, KENDALL A, CIPOLLA R. Segnet: a deep convolutional encoder-decoder architecture for image segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(12): 2481-2495.
[26] ZHAO H, SHI J, QI X, et al. Pyramid scene parsing network[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, Jul 21-26, 2017. Piscataway: IEEE, 2017: 2881-2890.
[27] CHEN L, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 40(4): 834-848.
[28] CHEN L, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation[J]. arXiv:1706.05587, 2017.
[29] CHEN L C, ZHU Y, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//Proceedings of the 15th European Conference on Computer Vision, Munich, Sep 8-14, 2018. Cham: Springer, 2018: 801-818.
[30] CHOLLET F. Xception: deep learning with depthwise separable convolutions[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition, Hawaii, Jul 21-26, 2017. Piscataway: IEEE, 2017: 1251-1258.
[31] WANG P, CHEN P, YUAN Y, et al. Understanding convolution for semantic segmentation[C]//Proceedings of the 2018 IEEE Winter Conference on Applications of Computer Vision, Nevada, Mar 12-15, 2018. Piscataway: IEEE, 2018: 1451-1460.
[32] ZHENG S, LU J, ZHAO H, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition,?Nashville, Jun 19-25, 2021. Piscataway: IEEE, 2021: 6881-6890.
[33] LIU Z, LIN Y, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition,?Nashville, Jun 19-25, 2021. Piscataway: IEEE, 2021: 10012-10022.
[34] LIU Y, ZHANG X, ZHANG S, et al. Part-aware prototype network for few-shot semantic segmentation[C]//Proceedings of the 16th European Conference on Computer Vision, Glasgow, Aug 23-28, 2020. Cham: Springer, 2020: 142-158.
[35] PASZKE A, GROSS S, MASSA F, et al. Pytorch: an imperative style, high-performance deep learning library[C]//Advances in Neural Information Processing Systems 32, 2019: 8024-8035.
[36] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, Jun 26-Jul 1, 2016. Piscataway: IEEE, 2016: 770-778.
[37] LI X, WEI T, CHEN Y P, et al. FSS-1000: a 1000-class dataset for few-shot segmentation[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, Jun 14-19, 2020. Piscataway: IEEE, 2020: 2869-2878.
[38] SHABAN A, BANSAL S, LIU Z, et al. One-shot learning for semantic segmentation[J]. arXiv:1709.03410, 2017.
[39] LIN T, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]//Proceedings of the 13th European Conference on Computer Vision, Zurich, Sep 6-12, 2014. Cham: Springer, 2014: 740-755.
[40] DENG J, DONG W, SOCHER R, et al. ImageNet: a large-scale hierarchical image database[C]//Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, Jun 20-26, 2009. Piscataway: IEEE, 2009: 248-255.
[41] LOSHCHILOV I, HUTTER F. Fixing weight decay regularization in Adam[C]//Proceedings of the 2018 IEEE International Conference on Learning Representations, Vancouver, Apr 30-May 3, 2018. Piscataway: IEEE, 2018.
[42] EVERINGHAM M, ESLAMI S M, VAN GOOL L, et al. The pascal visual object classes challenge: a retrospective[J]. International Journal of Computer Vision, 2015, 111(1): 98-136.
[43] HARIHARAN B, ARBELáEZ P, GIRSHICK R, et al. Simultaneous detection and segmentation[C]//Proceedings of the 13th European Conference on Computer Vision, Zurich, Sep 6-12, 2014. Cham: Springer, 2014: 297-312.
[44] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv:1409. 1556, 2014.
[45] RAKELLY K, SHELHAMER E, DARRELL T, et al. Few-shot segmentation propagation with guided networks[J]. arXiv: 1806. 07373, 2018.
[46] WANG H, ZHANG X, HU Y, et al. Few-shot semantic segmentation with democratic attention networks[C]//Proceedings of the 16th European Conference on Computer Vision,Glasgow, Aug 23-28, 2020. Cham: Springer, 2020: 730-746.
[47] RAKELLY K, SHELHAMER E, DARRELL T, et al. Conditional networks for few-shot semantic segmentation[C]//Proceedings of the 2018 IEEE International Conference on Learning Representations, Vancouver, Apr 30-May 3, 2018. Piscataway: IEEE, 2018.
[48] SIAM M, ORESHKIN B, JAGERSAND M. Adaptive masked proxies for few-shot segmentation[J]. arXiv:1902.11123, 2019.
[49] NGUYEN K, TODOROVIC S. Feature weighting and boosting for few-shot segmentation[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, Seoul, Oct 27-Nov 2, 2019. Piscataway: IEEE, 2019: 622-631.