RGB-D Image Saliency Detection Using Skip-Layer Convolutional Neural Network

doi:10.3778/j.issn.1002-8331.2008-0441

Abstract

Abstract: The saliency detection of RGB-D images aims to extract salient objects in three-dimensional images. In order to solve the problems of current saliency detection algorithms that are difficult to detect targets in light interference scenes and low-contrast targets, a saliency detection method for RGB-D images based on skip layer convolutional neural network is proposed. Firstly, the VGG network is adopted to separate the shallow and deep features of the RGB image and the depth image, and feature extraction is performed. Then the extracted features based on the skip-layer structure are connected to achieve the goal of fusing depth, color, high-level semantics and detailed information, and side outputs are generated. Finally, the side outputs are fused to obtain the best saliency detection map. Experimental results show that, compared with the deep supervised saliency detection and progressive complementary perception fusion saliency detection, the F value is improved by 0.095 3 and 0.060 6, and the average absolute error is reduced by 0.026 7 and 0.058 1, respectively.

Key words: saliency detection, convolutional neural network, skip-layer structure, deep learning, RGB-D

摘要： RGB-D图像显著性检测旨在提取三维图像中的显著目标。为解决当前显著性检测算法难以检测出光线干扰场景内的目标和低对比度的目标等问题，提出了基于跳层卷积神经网络的RGB-D图像显著性检测方法。利用VGG网络分离出RGB图像和深度图像的浅层与深层特征，而后进行特征提取；以跳层结构为基础连接提取到的特征，实现融合深度、颜色、高级语义和细节信息的目标，同时生成侧输出；将侧输出进行融合，得到最佳的显著性检测图。实验结果表明，相比于深度监督显著性检测和渐进式互补感知融合显著性检测方法，在F值指标上分别提高了0.095?3和0.060?6，在平均绝对误差指标上降低了0.026?7和0.058?1。

关键词: 显著性检测, 卷积神经网络, 跳层结构, 深度学习, RGB-D

CHEN Xitao, ZI Lingling, ZHANG Xueman. RGB-D Image Saliency Detection Using Skip-Layer Convolutional Neural Network[J]. Computer Engineering and Applications, 2022, 58(2): 252-258.

陈曦涛, 訾玲玲, 张雪曼. 采用跳层卷积神经网络的RGB-D图像显著性检测[J]. 计算机工程与应用, 2022, 58(2): 252-258.

References

[1] LI Z Z，ZHAO B J，TANG L B，et al.Ship classification based on convolutional neural networks[J].The Journal of Engineering，2019，2019（21）：7343-7346.
[2] SU W，WANG Z F.Widening residual refine edge reserved neural network for semantic segmentation[J].Multimedia Tools and Applications，2019，78（13）：18229-18247.
[3] WANG P，WANG C C，LAI J Y，et al.Traffic control approach based on multi-source data fusion[J].IET Intelligent Transport Systems，2019，13（5）：764-772.
[4] ZHOU T，LI Z X，ZHANG C L.Enhance the recognition ability to occlusions and small objects with robust faster R-CNN[J].International Journal of Machine Learning and Cybernetics，2019，10（11）：3155-3166.
[5] XIANG Y，LIU Z W，HUANG Y L，et al.Moving target detection with polarimetric distributed MIMO radar in heterogeneous clutter[J].The Journal of Engineering，2019，2019（21）：8009-8012.
[6] ZHANG X，HONG S G，NING A P，et al.Pedestrian detection with EDGE features of color image and HOG on depth images[J].Automatic Control and Computer Sciences，2020，54（2）：168-178.
[7] PENG H，LI B，XIONG W，et al.RGBD salient object detection：a benchmark and algorithms[C]//European Conference on Computer Vision，2014：92-109.
[8] CHENG Y，FU H，WEI X，et al.Depth enhanced saliency detection method[C]//Proceedings of International Conference on Internet Multimedia Computing and Service，2014：23-27.
[9] REN J Q，GONG X J，YU L，et al.Exploiting global priors for RGB-D saliency detection[C]//Conference on Computer Vision and Pattern Recognition Workshops（CVPRW），2015：25-32.
[10] WANG N，GONG X.Adaptive fusion for RGB-D salient object detection[J].IEEE Access，2019，7：55277-55284.
[11] HOU Q，CHENG M M，HU X W，et al.Deeply supervised salient object detection with short connections[C]//Conference on Computer Vision and Pattern Recognition（CVPR），2017：5300-5309.
[12] CHEN H，LI Y F.Progressively complementarity-aware fusion network for RGB-D salient object detection[C]//Conference on Computer Vision and Pattern Recognition（CVPR），2018：3051-3060.
[13] CHEN H，LI Y F，SU D.Multi-modal fusion network with multi-scale multipath and cross-modal interactions for RGB-D salient object detection[J].Pattern Recognition，2019，86（6）：376-385.
[14] HAN J W，CHEN H，LIU N，et al.CNNs-based RGB-D saliency detection via cross-view transfer and multiview fusion[J].IEEE Transactions on Systems，Man，and Cybernetics，2018，48（11）：3171-3183.
[15] CHEN H，LI Y.Three-stream attention-aware network for RGB-D salient object detection[J].IEEE Transactions on Image Processing，2019，28（6）：2825-2835.
[16] 李婧婧.基于深度学习的RGB-D显著性物体检测研究[D].大连：大连理工大学，2019.
LI J J.Research on RGB-D salient object detection based on deep learning[D].Dalian：Dalian University of Technology，2019.
[17] JU R，GE L，GENG W J，et al.Depth saliency based on anisotropic center-surround difference[C]//International Conference on Image Processing（ICIP），2014：1115-1119.
[18] HARIHARAN B，ARBELáEZ P，GIRSHICK R，et al.Hypercolumns for object segmentation and fine-grained localization[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：447-456.
[19] LONG J，SHELHAMER E，DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：3431-3440.
[20] XIE S，TU Z.Holistically-nested edge detection[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：1395-1403.
[21] 赵强，王爱平，刘政怡.背景与前景融合的RGB-D图像显著性检测[J].计算机科学与探索，2020，14（7）：1232-1242.
ZHAO Q，WANG A P，LIU Z Y.RGB-D image saliency detection via background and foreground fusion[J].Journal of Frontiers of Computer Science and Technology，2020，14（7）：1232-1242.
[22] 汪蕊，刘政怡，李炜.基于目标检索的RGB-D协同显著性研究[J].计算机工程与应用，2020，56（9）：228-233.
WANG R，LIU Z Y，LI W.Co-saliency detection based on objectness estimation for RGB-D image[J].Computer Engineering and Applications，2020，56（9）：228-233.
[23] 张守东，杨明，胡太.基于多特征融合的显著性目标检测算法[J].计算机科学与探索，2019，13（5）：834-845.
ZHANG S D，YANG M，HU T.Saliency object detection algorithm based on multi-feature fusion[J].Journal of Frontiers of Computer Science and Technology，2019，13（5）：834-845.
[24] 张志华，刘政怡.多角度融合的RGB-D显著检测[J].计算机工程与科学，2018，40（4）：681-689.
ZHANG Z H，LIU Z Y.Multi-angle fusion RGB-D saliency detection[J].Computer Engineering and Science，2018，40（4）：681-689.
[25] 吴建国，邵婷，刘政怡.融合显著深度特征的RGB-D图像显著目标检测[J].电子与信息学报，2017，39（9）：2148-2154.
WU J G，SHAO T，LIU Z Y.RGB-D salience detection based on integration feature of color and depth saliency map[J].Journal of Electronics & Information Technology，2017，39（9）：2148-2154.
[26] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409.
1556，2014.
[27] LI N，YE J，JI Y，et al.Saliency detection on light field[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2014：2806-2813.
[28] ZHU C，LI G.A three-pathway psychobiological framework of salient object detection using stereoscopic technology[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops，2017：3008-3014.