Weakly-Supervised Semantic Segmentation Method with Saliency Boundary Constraints

doi:10.3778/j.issn.1002-8331.2406-0116

Abstract

Abstract: In order to solve the problems of insufficient class activation and unclear pseudo-label boundaries in the existing weakly-supervised semantic segmentation methods, a weakly-supervised semantic segmentation method with saliency boundary constraints is proposed. A twin network with shared parameters is used as the class activation map generation network, and the images before and after the affine transformation are used as the inputs of two branches of the twin network. After obtaining different class activation maps, the complementary information is fused by consistency loss function to generate a more complete class activation map. The saliency correction module is designed, and boundary constraints are introduced into the class activation map to suppress the wrong activation of background information. At the same time, the saliency affinity module is designed to learn the affinity matrix between pixels from the saliency map, which further refines the initial pseudo-labels and improves the semantic segmentation performance of the model. The experimental results show that the mIoU value of this method is 71.4% on PASCAL VOC 2012 validation set, and the performance is improved by 2.1 percentage points compared with the baseline, and the mIoU value on the test set is 70.8%. The mIoU value on the COCO 2014 validation set is 39.2%, which shows a good segmentation result, and the method can better complete the task of weakly-supervised semantic segmentation.

Key words: weakly-supervised semantic segmentation, image-level labels, Transformer, convolutional neural network, siamese network, saliency map

摘要： 针对现有弱监督语义分割方法存在的类激活不足、伪标签边界不清晰的问题，提出了融合显著边界约束的弱监督语义分割方法。提出由共享参数的孪生网络作为类激活图生成网络，将仿射变换前后的图像作为孪生网络两个分支的输入，得到不同的类激活图后，通过一致性损失函数融合仿射变换前后的互补信息，以生成具有完整信息的类激活图。设计显著性修正模块，在类激活图中引入边界约束，抑制背景信息的错误激活；同时，设计显著性亲和模块从显著图中学习像素之间的亲和矩阵，进一步细化初始伪标签，提升模型的语义分割性能。实验结果表明，该方法在PASCAL VOC 2012验证集上的mIoU值为71.4%，与基线相比，性能提升了2.1个百分点，测试集上的mIoU值为70.8%；在COCO 2014验证集上的mIoU值为39.2%，展现了良好的分割结果，该方法可以更好地完成弱监督语义分割任务。

关键词: 弱监督语义分割, 图像级标签, Transformer, 卷积神经网络, 孪生网络, 显著图

BAI Xuefei, ZHANG Lina, WANG Wenjian. Weakly-Supervised Semantic Segmentation Method with Saliency Boundary Constraints[J]. Computer Engineering and Applications, 2025, 61(19): 214-225.

白雪飞, 张丽娜, 王文剑. 融合显著边界约束的弱监督语义分割方法[J]. 计算机工程与应用, 2025, 61(19): 214-225.

References

[1] BEARMAN A, RUSSAKOVSKY O, FERRARI V, et al. What’s the point: semantic segmentation with point supervision[C]//Proceedings of the 14th European Conference on Computer Vision. Cham: Springer, 2016: 549-565.
[2] DAI J F, HE K M, SUN J. BoxSup: exploiting bounding boxes to supervise convolutional networks for semantic segmentation[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway: IEEE, 2015: 1635-1643.
[3] LIN D, DAI J F, JIA J Y, et al. ScribbleSup: scribble-supervised convolutional networks for semantic segmentation[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 3159-3167.
[4] YIN X, IM W, MIN D B, et al. Fine-grained background representation for weakly supervised semantic segmentation[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2024, 34(11): 11739-11750.
[5] PINHEIRO P O, COLLOBERT R. From image-level to pixel-level labeling with Convolutional Networks[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2015: 1713-1721.
[6] PAPANDREOU G, CHEN L C, MURPHY K P, et al. Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway: IEEE, 2015: 1742-1750.
[7] CHEN Q, YANG L X, LAI J H, et al. Self-supervised image-specific prototype exploration for weakly supervised semantic segmentation[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 4278-4288.
[8] JIANG P T, YANG Y Q, HOU Q B, et al. L2G: a simple local-to-global knowledge transfer framework for weakly supervised semantic segmentation[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 16865-16875.
[9] LI R W, MAI Z D, ZHANG Z B, et al. TransCAM: transformer attention-based CAM refinement for weakly supervised semantic segmentation[J]. Journal of Visual Communication and Image Representation, 2023, 92: 103800.
[10] XU L, OUYANG W L, BENNAMOUN M, et al. Multi-class token transformer for weakly supervised semantic segmentation[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 4300-4309.
[11] AHMADI R, KASAEI S. Leveraging swin transformer for local-to-global weakly supervised semantic segmentation[C]//Proceedings of the 2024 13th Iranian/3rd International Machine Vision and Image Processing Conference. Piscataway: IEEE, 2024: 1-7.
[12] WU Y C, YE X C, YANG K Q, et al. DuPL: dual student with trustworthy progressive learning for robust weakly supervised semantic segmentation[C]//Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2024: 3534-3543.
[13] PENG Z L, HUANG W, GU S Z, et al. Conformer: local features coupling global representations for visual recognition[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021: 357-366.
[14] KIM D, CHO D, YOO D, et al. Two-phase learning for weakly supervised object localization[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway: IEEE, 2017: 3554-3563.
[15] LEE S, LEE M, LEE J, et al. Railroad is not a train: saliency as pseudo-pixel supervision for weakly supervised semantic segmentation[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 5491-5501.
[16] ZHOU B L, KHOSLA A, LAPEDRIZA A, et al. Learning deep features for discriminative localization[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 2921-2929.
[17] WEI Y C, FENG J S, LIANG X D, et al. Object region mining with adversarial erasing: a simple classification to semantic segmentation approach[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 6488-6496.
[18] JIANG P T, HOU Q B, CAO Y, et al. Integral object mining via online attention accumulation[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2019: 2070-2079.
[19] LEE J, KIM E, LEE S, et al. FickleNet: weakly and semi-supervised semantic image segmentation using stochastic inference[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 5262-5271.
[20] WANG Y D, ZHANG J, KAN M N, et al. Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 12272-12281.
[21] ZHAO X Q, TANG F L, WANG X Y, et al. SFC: shared feature calibration in weakly supervised semantic segmentation[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2024, 38(7): 7525-7533.
[22] CHEN T, YAO Y Z, HUANG X G, et al. Spatial structure constraints for weakly supervised semantic segmentation[J]. IEEE Transactions on Image Processing, 2024, 33: 1136-1148.
[23] FAN J S, ZHANG Z X, TAN T N, et al. CIAN: cross-image affinity net for weakly supervised semantic segmentation[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 10762-10769.
[24] CHEN L Y, WU W W, FU C C, et al. Weakly supervised semantic segmentation with boundary exploration[C]//Proceedings of the 16th European Conference on Computer Vision. Cham: Springer, 2020: 347-362.
[25] FAN J S, ZHANG Z X, TAN T N. Employing multi-estimations for weakly-supervised semantic segmentation[C]//Proceedings of the 16th European Conference on Computer Vision. Cham: Springer, 2020: 332-348.
[26] HUO D G, SU Y K, WU QY. Dual progressive transformations for weakly supervised semantic segmentation[C]//Proceedings of 2022 IEEE Conference on Computer Vision and Pattern Recognition, 2022: 4278-4288.
[27] PENG Z L, WANG G C, XIE L X, et al. USAGE: a unified seed area generation paradigm for weakly supervised semantic segmentation[C]//Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2023: 624-634.
[28] DENG J L, LI Z H. 2023. Dual-augmented Transformer network for weakly supervised semantic segmentation[J]. arXiv:2310.00307, 2023.
[29] WANG X, YOU S D, LI X, et al. Weakly-supervised semantic segmentation by iteratively mining common object features[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 1354-1362.
[30] HOU Q B, JIANG P T, WEI Y C, et al. Self-erasing network for integral object attention[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook, USA: Curran Associates Inc, 2018: 547-557.
[31] FAN R C, HOU Q B, CHENG M M, et al. Associating inter-image salient instances for weakly supervised semantic segmentation[C]//Proceedings of the 15th European Conference on Computer Vision. Cham: Springer, 2018: 371-388.
[32] YAO Q, GONG X J. Saliency guided self-attention network for weakly and semi-supervised semantic segmentation[J]. IEEE Access, 2020, 8: 14413-14423.
[33] XU L, BENNAMOUN M, BOUSSAID F, et al. Auxiliary tasks enhanced dual-affinity learning for weakly supervised semantic segmentation[J]. IEEE Transactions on Neural Networks and Learning Systems, 2025, 36(3): 5082-5096.
[34] HARIHARAN B, ARBELáEZ P, BOURDEV L, et al. Semantic contours from inverse detectors[C]//Proceedings of the 2011 International Conference on Computer Vision. Piscataway: IEEE, 2011: 991-998.
[35] DENG J, DONG W, SOCHER R, et al. ImageNet: a large-scale hierarchical image database[C]//Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2009: 248-255.
[36] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848.
[37] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]//Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition, 2015: 3431-3440.
[38] LEE J, KIM E, YOON S. Anti-adversarially manipulated attributions for weakly and semi-supervised semantic segmentation[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 4070-4078.
[39] YAO Y Z, CHEN T, XIE G S, et al. Non-salient region object mining for weakly supervised semantic segmentation[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 2623-2632.
[40] XU R T, WANG C W, SUN J X, et al. Self correspondence distillation for end-to-end weakly-supervised semantic segmentation[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2023, 37(3): 3045-3053.
[41] RU L X, ZHAN Y B, YU B S, et al. Learning affinity from attention: end-to-end weakly-supervised semantic segmentation with transformers[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 16825-16834.
[42] AHN J, CHO S, KWAK S. Weakly supervised learning of instance segmentation with inter-pixel relations[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 2204-2213.
[43] SU Y K, DENG J L, LI Z H. COMNet: co-occurrent matching for weakly supervised semantic segmentation[J]. arXiv:2309.
16959, 2023.
[44] ZHAI W, WU P Y, ZHU K, et al. Background activation suppression for weakly supervised object localization[C]//Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 14228-14237.
[45] ZHANG B F, XIAO J M, WEI Y C, et al. Reliability does matter: an end-to-end weakly supervised semantic segmentation approach[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 12765-12772.
[46] FAN J S, ZHANG Z X, SONG C F, et al. Learning integral objects with intra-class discriminator for weakly-supervised semantic segmentation[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 4282-4291.
[47] WU T, HUANG J S, GAO G Y, et al. Embedded discriminative attention mechanism for weakly supervised semantic segmentation[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 16760-16769.
[48] LIU Y, WU Y H, WEN P S, et al. Leveraging instance-, image- and dataset-level information for weakly supervised instance segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(3): 1415-1428.
[49] SUN W X, ZHANG J, BARNES N. Inferring the class conditional response map for weakly supervised semantic segmentation[C]//Proceedings of the 2022 IEEE/CVF Winter Conference on Applications of Computer Vision. Piscataway: IEEE, 2022: 2653-2662.
[50] LI Y, KUANG Z H, LIU L Y, et al. Pseudo-mask matters in weakly-supervised semantic segmentation[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2021: 6944-6953.
[51] HUANG Z L, WANG X G, WANG J S, et al. Weakly-supervised semantic segmentation network with deep seeded region growing[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 7014-7023.
[52] ZHOU T F, ZHANG M J, ZHAO F, et al. Regional semantic contrast and aggregation for weakly supervised semantic segmentation[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 4289-4299.