Boundary-Aware Real-Time Semantic Segmentation Network

doi:10.3778/j.issn.1002-8331.2112-0247

Abstract

Abstract: Currently most real-time semantic segmentation networks not only process detailed information such as boundaries and textures, but also ignore the features of semantic boundary regions which leads to the deterioration of the quality of object boundary segmentation. To solve this problem, this paper proposes a boundary-aware segmentation network, which mainly improves the quality of boundary semantic segmentation from three aspects. Firstly, a boundary-aware learning mechanism is proposed to use location information to reduce the coupling degree of boundary features and details near the contours, so that boundary perception and location relationships can mutually reinforce each other. Secondly, a lightweight region adaptive module is designed to enhance the ability of convolution network to model complex semantic boundary regions. Finally, according to the different pixel contribution values of the sampling area, an efficient distinctive atrous spatial pyramid pooling module is designed to enhance important details and semantic features. In experiment, compared with the benchmark, the accuracy on the Cityscapes validation set is improved by about 5.8?percentage points, and the accuracy on the Cityscapes test set is 74.9% with an inference speed of 47.2 FPS. Compared with BiSeNetV2 algorithm on the CamVid data set, the mIoU is improved by about 3.96 percentage points.

Key words: image processing, semantic segmentation, boundary aware, semantic boundary

摘要： 目前多数实时语义分割网络不仅同时处理边界和纹理等细节信息而且还忽略了语义边界区域特征，从而导致物体边界分割质量下降。针对该问题，提出一种边界感知的实时语义分割网络，主要从三个方面提高边界语义分割质量。提出了边界感知学习机制利用位置信息降低边界特征和轮廓附近细节的耦合度使边界感知和位置关系相互促进。设计轻量级区域自适应模块增强卷积网络对复杂语义边界区域的建模能力。根据采样区域像素贡献值不同设计了高效的空洞空间金字塔池化模块以增强重要的细节和语义特征。实验方面，与基准相比，在Cityscapes验证集上精度提升了约5.8个百分点，在Cityscapes测试集上以47.2 FPS的推理速度使精度达到了74.9%。在CamVid数据集上与BiSeNetV2算法相比mIoU提升了约3.96个百分点。

关键词: 图像处理, 语义分割, 边界感知, 语义边界

HUO Zhanqiang, JIA Haiyang, QIAO Yingxu, LUO Fen, CHEN Wei. Boundary-Aware Real-Time Semantic Segmentation Network[J]. Computer Engineering and Applications, 2022, 58(17): 165-173.

霍占强, 贾海洋, 乔应旭, 雒芬, 陈玮. 边界感知的实时语义分割网络[J]. 计算机工程与应用, 2022, 58(17): 165-173.

References

[1] 徐辉，祝玉华，甄彤，等.深度神经网络图像语义分割方法综述[J].计算机科学与探索，2021，15（1）：47-59.
XU H，ZHU Y H，ZHEN T，et al.Survey of image semantic segmentation methods based on deep neural network[J].Journal of Frontiers of Computer Science and Technology，2021，15（1）：47-59.
[2] 祁欣，袁非牛，史劲亭，等.多层次特征融合网络的语义分割算法[J/OL].计算机科学与探索：1-13[2021-12-10].http：//kns.cnki.net/kcms/detail/11.5602.TP.20211014.1148.
002.html.
QI X，YUAN F N，SHI J T，et al.Semantic segmentation algorithm of multi-level feature fusion network[J/OL].Journal of Frontiers of Computer Science and Technology：1-13[2021-12-10].http：//kns.cnki.net/kcms/detail/11.5602.TP.20211014.1148.002.html.
[3] LI L，QIAN B，LIAN J，et al.Traffic scene segmentation based on RGB-D image and deep learning[J].IEEE Transactions on Intelligent Transportation Systems，2017，19（5）：1664-1669.
[4] ZHOU W，WORRALL S，ZYNER A，et al.Automated process for incorporating drivable path into real-time semantic segmentation[C]//2018 IEEE International Conference on Robotics and Automation（ICRA），2018：6039-6044.
[5] FU J，LIU J，TIAN H，et al.Dual attention network for scene segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：3146-3154.
[6] DAVIS L S，ROSENFELD A，WESZKA J S.Region extraction by averaging and thresholding[J].IEEE Transactions on Systems，Man，and Cybernetics，1975（3）：383-388.
[7] TAO W，JIN H，ZHANG Y.Color image segmentation based on mean shift and normalized cuts[J].IEEE Transactions on Systems，Man，and Cybernetics，Part B：Cybernetics，2007，37（5）：1382-1389.
[8] MUTHUKRISHNAN R，RADHA M.Edge detection techniques for image segmentation[J].International Journal of Computer Science & Information Technology，2011，3（6）：259.
[9] SHI J，MALIK J.Normalized cuts and image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2000，22（8）：888-905.
[10] 赵小强，徐慧萍.分级特征融合的图像语义分割[J].计算机科学与探索，2021，15（5）：949-957.
ZHAO X Q，XU H P.Image semantic segmentation method with hierarchical feature fusion[J].Journal of Frontiers of Computer Science and Technology，2021，15（5）：949-957.
[11] LONG J，SHELHAMER E，DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：3431-3440.
[12] CHEN L C，ZHU Y，PAPANDREOU G，et al.Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：801-818.
[13] YU C，GAO C，WANG J，et al.BiSeNet V2：bilateral network with guided aggregation for real-time semantic segmentation[J].arXiv：2004.02147，2020.
[14] YIN M，YAO Z，CAO Y，et al.Disentangled non-local neural networks[C]//European Conference on Computer Vision.Cham：Springer，2020：191-207.
[15] FAN M，LAI S，HUANG J，et al.Rethinking BiSeNet for real-time semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：9716-9725.
[16] ZHU L，JI D，ZHU S，et al.Learning statistical texture for semantic segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：12537-12546.
[17] HOU Q，ZHOU D，FENG J.Coordinate attention for efficient mobile network design[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：13713-13722.
[18] GAO S，CHENG M M，ZHAO K，et al.Res2net：a new multi-scale backbone architecture[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2021，43（2）：652-662.
[19] SZEGEDY C，LIU W，JIA Y，et al.Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：1-9.
[20] CHOLLET F.Xception：deep learning with depthwise separable convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：1251-1258.
[21] DONG J，YUAN J，LI L，et al.An efficient semantic segmentation method using pyramid ShuffleNet V2 with vortex pooling[C]//2019 IEEE 31st International Conference on Tools with Artificial Intelligence（ICTAI），2019：1214-1220.
[22] DONG G，YAN Y，SHEN C，et al.Real-time high-performance semantic image segmentation of urban street scenes[J].IEEE Transactions on Intelligent Transportation Systems，2020，22（6）：3258-3274.
[23] CORDTS M，OMRAN M，RAMOS S，et al.The cityscapes dataset for semantic urban scene understanding[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：3213-3223.
[24] BROSTOW G J，SHOTTON J，FAUQUEUR J，et al.Segmentation and recognition using structure from motion point clouds[C]//European Conference on Computer Vision.Berlin，Heidelberg：Springer，2008：44-57.
[25] CHEN L C，PAPANDREOU G，KOKKINOS I，et al.Deeplab：semantic image segmentation with deep convolutional nets，atrous convolution，and fully connected crfs[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，40（4）：834-848.
[26] ZHAO H，SHI J，QI X，et al.Pyramid scene parsing network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：2881-2890.
[27] PASZKE A，CHAURASIA A，KIM S，et al.Enet：a deep neural network architecture for real-time semantic segmentation[J].arXiv：1606.02147，2016.
[28] ROMERA E，ALVAREZ J M，BERGASA L M，et al.Erfnet：efficient residual factorized convnet for real-time semantic segmentation[J].IEEE Transactions on Intelligent Transportation Systems，2017，19（1）：263-272.
[29] MAZZINI D.Guided upsampling network for real-time semantic segmentation[J].arXiv：1807.07466，2018.
[30] YU C，WANG J，PENG C，et al.Bisenet：bilateral segmentation network for real-time semantic segmentation[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：325-341.
[31] ZHAO H，QI X，SHEN X，et al.Icnet for real-time semantic segmentation on high-resolution images[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：405-420.
[32] MEHTA S，RASTEGARI M，SHAPIRO L，et al.Espnetv2：a light-weight，power efficient，and general purpose convolutional neural network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：9190-9200.
[33] POUDEL R P K，LIWICKI S，CIPOLLA R.Fast-scnn：fast semantic segmentation network[J].arXiv：1902.04502，2019.
[34] BADRINARAYANAN V，KENDALL A，CIPOLLA R.Segnet：a deep convolutional encoder-decoder architecture for image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，39（12）：2481-2495.
[35] WANG H，JIANG X，REN H，et al.SwiftNet：real-time video object segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：1296-1305.