Research on Propagation of Similarity Between Frames in Video Object Segmentation

doi:10.3778/j.issn.1002-8331.2009-0446

Abstract

Abstract: At present, video target segmentation algorithms are mostly based on matching and propagation strategies to segment the target, and often use the information of the previous frame in the form of mask or optical flow. A new way of feature propagation method between frames is explored in this paper. It uses short-term matching module to extract the previous frame information for propagation, and proposes an object segmentation model for video sequence data. Through the long-term matching module and the short-term matching module to perform pixel-level matching with the first frame and the previous frame respectively, the global similarity map and the local similarity map are obtained, as well as the mask of the previous frame and the feature map of the current frame. The result is obtained through the segmentation network after passing through two optimized networks. Experiments on the public video object segmentation dataset show that the proposed model achieves good regional similarity and contour accuracy J&F 86.5% and 77.4% on single target and multi-target segmentation respectively, without online fine-tuning, and 21 frames per second can be calculated. The short-term matching module proposed in this paper is more conducive to extracte the information of the previous frame than simply using the mask. Through the combination of the long-term matching module and the short-term matching module, efficient video target segmentation can be achieved without online fine-tuning, which is more suitable applied to mobile robot environment perception content.

Key words: visual perception, video object segmentation, feature propagation, long-short-term matching

摘要： 目前视频目标分割算法多是基于匹配和传播策略分割目标，常常以掩模或者光流的方式利用前一帧的信息，探索了新的帧间特征传播方式，利用短时匹配模块提取前一帧信息并传播给当前帧，提出一种面向视频序列数据的目标分割模型。通过长时匹配模块和短时匹配模块分别与第一帧和前一帧做相关操作进行像素级匹配，得到的全局相似性图和局部相似性图，以及前一帧的掩模和当前帧的特征图，经过两个优化网络后通过分割网络得到分割结果。在视频目标分割公开数据集上的实验表明，所提出方法在单目标和多目标上分别取得了86.5%和77.4%的区域相似度和轮廓精度均值，每秒可计算21帧。提出的短时匹配模块比仅使用掩模更有利于提取前一帧的信息，通过长时匹配模块和短时匹配模块的结合，不使用在线微调即可实现高效的视频目标分割，适合应用于移动机器人视觉感知。

关键词: 视觉感知, 视频目标分割, 特征传播, 长-短时匹配

ZHANG Xuerui, SUN Fengming, YUAN Xia. Research on Propagation of Similarity Between Frames in Video Object Segmentation[J]. Computer Engineering and Applications, 2022, 58(6): 227-233.

章雪瑞, 孙凤铭, 袁夏. 视频目标分割中帧间相似性传播的研究[J]. 计算机工程与应用, 2022, 58(6): 227-233.

References

[1] CAELLES S，MANINIS K K，PONT-TUSET J，et al.One-shot video object segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Hawaii：IEEE，2017：5320-5329.
[2] HU Y T，HUANG J B，SCHWING A G.VideoMatch：matching based video object segmentation[C]//Proceedings of the European Conference on Computer Vision（ECCV）.Munich：Springer，2018：56-73.
[3] KHOREVA A，PERAZZI F，BENENSON R，et al.Learning video object segmentation from static images[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Hawaii：IEEE，2017：3491-3500.
[4] LI X X，LOY C C.Video object segmentation with joint re-identification and attention-aware mask[C]//Proceedings of the European Conference on Computer Vision（ECCV）.Munich：Springer，2018：93-110.
[5] OH S W，LEE J Y，SUNKAVALLI K，et al.Fast video object segmentation by reference-guided mask propagation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Salt Lake City：IEEE，2018：7376-7385.
[6] CHENG J C，TSAI Y H，HUNG W C，et al.Fast and accurate online video object segmentation via tracking parts[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Salt Lake City：IEEE，2018：7415-7424.
[7] YANG L J，WANG Y R，XIONG X H，et al.Efficient video object segmentation via network modulation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Salt Lake City：IEEE，2018：6499-6507.
[8] KHOREVA A，BENENSON R，ILG E，et al.Lucid data dreaming for video object segmentation[J].International Journal of Computer Vision，2019，127：1175-1197.
[9] LI J，HAN K，WANG P，et al.Anisotropic convolutional networks for 3D semantic scene completion[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Seattle：IEEE，2020：3348-3356.
[10] GAO S H，CHENG M M，ZHAO K，et al.Res2Net：a new multi-scale backbone architecture[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2019，43（2）：652-662.
[11] LIN T Y，DOLLAR P，GIRSHICK R，et al.Feature pyramid networks for object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Hawaii：IEEE，2017：936-944.
[12] LI B，YAN J J，WU W，et al.High performance visual tracking with siamese region proposal network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Salt Lake City：IEEE，2018：8971-8980.
[13] WANG Z Q，XU J，LIU L，et al.RANet：ranking attention network for fast video object segmentation[C]//Proceedings of the IEEE International Conference on Computer Vision（ICCV）.Seoul：IEEE，2019：3977-3986.
[14] ILG E，MAYER N，SAIKIA T，et al.FlowNet 2.0：evolution of optical flow estimation with deep networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Hawaii：IEEE，2017：1647-1655.
[15] PERAZZI F，PONT-TUSET J，MCWILLIAMS B，et al.A benchmark dataset and evaluation methodology for video object segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Las Vegas：IEEE，2016：724-732.
[16] PONT-TUSET J，PERAZZI F，CAELLES S，et al.The 2017 DAVIS challenge on video object segmentation[J].arXiv：1704.00675，2017.
[17] XU N，YANG L，FAN Y，et al.YouTube-VOS：sequence-to-sequence video object segmentation[C]//Proceedings of the European Conference on Computer Vision（ECCV）.Munich：Springer，2018：585-601.
[18] DENG J，DONG W，SOCHER R，et al.ImageNet：a large-scale hierarchical image database[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition（CVPR）.Miami：IEEE，2009：248-255.
[19] LIN T Y，GOYAL P，GIRSHICK R，et al.Focal loss for dense object detection[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2018，42（2）：318-327.
[20] LUITEN J，VOIGTLAENDER P，LEIBE B.PReMVOS：proposal-generation，refinement and merging for video object segmentation[C]//Asian Conference on Computer Vision（ACCV）.Perth：Springer，2018：565-580.