RGB-D Saliency Detection Based on Multi-Level Feature Fusion

doi:10.3778/j.issn.1002-8331.2112-0219

Abstract

Abstract: When most RGB-D saliency detection methods explore the cross-modal information of each layer, they often directly merge the depth map with the RGB map without processing, and use the same fusion strategy at each level. However, this will cause two problems：（1） Low-quality depth maps will bring a lot of redundant information into the network, which will have a negative impact on detection; （2） The same fusion strategy is adopted at all levels, which ignores that the model has different degrees of attention to global and local features at different levels. In order to solve the above problems, a top-down multi-level feature fusion structure is proposed. The low-quality depth map information is effectively filtered through the design of the depth enhancement module. The high-level fusion module is designed to effectively integrate the global features in the high-level. The low-level fusion module is designed to effectively extract and merge useful local features. Comprehensive experiments on five public datasets and seven advanced models show that the method in this paper has superiority in the four indicators of F value, avgF value, S value and MAE.

Key words: saliency object detection, cross-modal feature, channel attention, feature blending

摘要： 大多数RGB-D显著性检测方法在探索各层跨模态信息时，往往直接将深度图不加处理地与RGB图进行融合，并且在各个层次采用相同的融合策略。然而，这会产生两个问题：（1）低质量深度图会把大量的冗余信息带入网络中，给检测带来负面影响；（2）在各个层次上采用相同的融合策略，忽略了模型在不同层次对全局和局部特征具有不同的关注度。为了解决上述问题，提出了一种自顶向下的多层次特征融合结构，通过设计深度增强模块有效地过滤低质量深度图信息；设计高层交融模块有效地整合高层中的全局特征；设计低层交融模块有效提取与融合有用的局部特征。通过在5个公共数据集上与7种先进模型进行的综合实验表明，该模型在F（F-measure）指标、avgF（average F-measure）指标、S（S-measure）指标和MAE（mean absolute error）4个指标上表现出了良好的性能。

关键词: 显著性检测, 跨模态特征, 通道注意力, 特征交融

SHI Yue, YU Wanjun, CHEN Ying. RGB-D Saliency Detection Based on Multi-Level Feature Fusion[J]. Computer Engineering and Applications, 2023, 59(7): 207-213.

史悦, 于万钧, 陈颖. 基于多层次特征融合的RGB-D显著性检测[J]. 计算机工程与应用, 2023, 59(7): 207-213.

References

[1] CHENG Y，FU H，WEI X，et al.Depth enhanced saliency detection method[C]//Proceedings of International Conference on Internet Multimedia Computing and Service，2014：23-27.
[2] CONG R，LEI J，ZHANG C，et al.Saliency detection for stereoscopic images based on depth confidence analysis and multiple cues fusion[J].IEEE Signal Processing Letters，2016，23（6）：819-823.
[3] ZHAO J X，CAO Y，FAN D P，et al.Contrast prior and fluid pyramid integration for RGBD salient object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：3927-3936.
[4] HAN J，CHEN H，LIU N，et al.CNNs-based RGB-D saliency detection via cross-view transfer and multiview fusion[J].IEEE Transactions on Cybernetics，2017，48（11）：3171-3183.
[5] PIAO Y，JI W，LI J，et al.Depth-induced multi-scale recurrent attention network for saliency detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：7254-7263.
[6] CHEN H，LI Y.Progressively complementarity-aware fusion network for RGB-D salient object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：3051-3060.
[7] CIPTADI A，HERMANS T，REHG J M.An in depth view of saliency[C]//Proceedings of Georgia Institute of Technology，2013.
[8] FENG D，BARNES N，YOU S，et al.Local background enclosure for RGB-D salient object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：2343-2350.
[9] REN J，GONG X，YU L，et al.Exploiting global priors for RGB-D saliency detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops，2015：25-32.
[10] QU L，HE S，ZHANG J，et al.RGBD salient object detection via deep fusion[J].IEEE Transactions on Image Processing，2017，26（5）：2274-2285.
[11] LIU Z，SHI S，DUAN Q，et al.Salient object detection for RGB-D image by single stream recurrent convolution neural network[J].Neurocomputing，2019，363：46-57.
[12] CHEN H，LI Y.Three-stream attention-aware network for RGB-D salient object detection[J].IEEE Transactions on Image Processing，2019，28（6）：2825-2835.
[13] FU K，FAN D P，JI G P，et al.JL-DCF：joint learning and densely-cooperative fusion framework for RGB-D salient object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：3052-3062.
[14] LI G，LIU Z，YE L，et al.Cross-modal weighting network for RGB-D salient object detection[C]//Proceedings of 16th European Conference on Computer Vision，2020：665-681.
[15] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[16] CHEN L C，PAPANDREOU G，KOKKINOS I，et al.DeepLAB：semantic image segmentation with deep convolutional nets，atrous convolution，and fully connected CRFS[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，40（4）：834-848.
[17] WOO S，PARK J，LEE J Y，et al.CBAM：convolutional block attention module[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：3-19.
[18] PENG H，LI B，XIONG W，et al.RGBD salient object detection：a benchmark and algorithms[C]//Proceedings of the European Conference on Computer Vision，2014：92-109.
[19] NIU Y，GENG Y，LI X，et al.Leveraging stereopsis for saliency analysis[C]//Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition，2012：454-461.
[20] ZHU C，LI G.A three-pathway psychobiological framework of salient object detection using stereoscopic technology[C]//Proceedings of the IEEE International Conference on Computer Vision Workshops，2017：3008-3014.
[21] FAN D P，LIN Z，ZHANG Z，et al.Rethinking RGB-D salient object detection：models，data sets，and large-scale benchmarks[J].IEEE Transactions on Neural Networks and Nian Liu，Systems，2020，32（5）：2075-2089.
[22] LI G，LIU Z，LING H.ICNet：information conversion network for RGB-D based salient object detection[J].IEEE Transactions on Image Processing，2020，29：4873-4884.
[23] ZHAO X，ZHANG L，PANG Y，et al.A single stream network for robust and real-time RGB-D salient object detection[C]//Proceedings of the European Conference on Computer Vision，2020：646-662.
[24] CHEN C，WEI J，PENG C，et al.Depth-quality-aware salient object detection[J].IEEE Transactions on Image Processing，2021，30：2350-2363.
[25] LIU N，ZHANG N，HAN J.Learning selective self-mutual attention for RGB-D saliency detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：13756-13765.