Online Category-Free Point-Wise Multi-Object Tracking and Segmentation

doi:10.3778/j.issn.1002-8331.2103-0044

Abstract

Abstract: Most multi-object tracking algorithms based on deep learning perform tracking by bounding boxes predict from object detection algorithm. However, when objects are partially occluded, their bounding boxes are prone to overlap, which greatly affects the tracking results. To address the problem, a novel online category-free point-wise multi-object tracking and segmentation method（CPMOTS） is proposed. It adopts pixel-level instance segmentation masks rather than the bounding-box-based object representation. CPMOTS uses a parallel structure to simultaneously segment and track multi-category objects, and guarantees the operating efficiency, which is quite practical in real scenarios. CPMOTS first obtains instance masks from instance segmentation network and samples unordered 2D point set from the masks, then obtains discriminative instance embeddings from the corresponding points features. Finally, an intuitive and effective attention module is utilized to explicitly model interdependencies between channels, through which it can adaptively learn the importance of each feature channel. Therefore, CPMOTS can selectively emphasize informative features and suppress less useful ones to achieve feature re-calibration, which boosts the performance of the network. Evaluations on KITTI MOTS dataset show that the CPMOTS outperforms many previous works with a near real-time speed 16 frame/s.

Key words: deep learning, multi-object tracking and segmentation, instance segmentation, attention module, feature re-calibration

摘要： 现有基于深度学习的多目标跟踪算法大多利用目标检测任务预测的边界框跟踪目标，当目标间存在遮挡时，边界框会产生重叠进而影响跟踪准确度，针对这个问题，提出了一种在线多类别逐点式多目标跟踪与分割（category-free point-wise multi-object tracking and segmentation，CPMOTS）算法。该算法摒弃了边界框的目标表征方式，利用实例分割的像素级掩码表征目标进行跟踪，网络采用并行结构同时分割与跟踪多类别目标，并保证了运行效率，这在真实场景中有很强的实用性。CPMOTS首先利用实例分割网络得到实例分割掩码，对其采样得到无序点集；然后将点集的特征输入跟踪网络得到判别性的实例级嵌入向量；最后将该嵌入向量通过直观高效的注意力模块以显式建模其通道间的依赖关系，自适应学习每个特征通道的重要程度，依照这个重要程度选择性地强化有用的特征，抑制无用的特征，实现通道特征重标定，从而提高算法的性能。在多目标跟踪与分割基准数据集KITTI MOTS的实验表明，CPMOTS跟踪的精度优于大部分其他对比方法，并达到了16 frame/s的近实时速度。

关键词: 深度学习, 多目标跟踪与分割, 实例分割, 注意力模块, 特征重标定

BI Xin, TAN Jingang, ZHANG Guanghui. Online Category-Free Point-Wise Multi-Object Tracking and Segmentation[J]. Computer Engineering and Applications, 2022, 58(19): 193-201.

毕鑫, 谭锦钢, 张广慧. 在线多类别逐点式多目标跟踪与分割[J]. 计算机工程与应用, 2022, 58(19): 193-201.

References

[1] 李俊彦，宋焕生，张朝阳，等.基于视频的多目标车辆跟踪及轨迹优化[J].计算机工程与应用，2020，56（5）：194-199.
LI J Y，SONG H S，ZHANG Z H，et al.Multi-object vehicle tracking and trajectory optimization based on video[J].Computer Engineering and Applications，2020，56（5）：194-199.
[2] 单玉刚，胡卫国.尺度方向自适应视觉目标跟踪方法综述[J].计算机工程与应用，2020，56（9）：13-23.
SHAN Y G，HU W G.Review of visual object tracking algorithms of adaptive direction and scale[J].Computer Engineering and Applications，2020，56（9）：13-23.
[3] XU J，CAO Y，ZHANG Z，et al.Spatial-temporal relation networks for multi-object tracking[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：3988-3998.
[4] BERGMANN P，MEINHARDT T，LEAL-TAIXE L.Tracking without bells and whistles[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：941-951.
[5] KEUPER M，TANG S，ANDRES B，et al.Motion segmentation & multiple object tracking by correlation co-clustering[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2018，42（1）：140-153.
[6] TANG S，ANDRILUKA M，ANDRES B，et al.Multiple people tracking by lifted multi cut and person re-identification[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：3539-3548.
[7] HORNAKOVA A，HENSCHEL R，ROSENHAHN B，et al.Lifted disjoint paths with application in multiple object tracking[C]//Proceedings of the International Conference on Machine Learning，2020：4364-4375.
[8] BRASó G，LEAL-TAIXé L.Learning a neural solver for multiple object tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：6247-6257.
[9] LI B，WU W，WANG Q，et al.SiamRPN++：Evolution of Siamese visual tracking with very deep networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：4282-4291.
[10] LI B，YAN J，WU W，et al.High performance visual tracking with siamese region proposal network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：8971-8980.
[11] ZHANG Z，PENG H.Deeper and wider siamese networks for real?time visual tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：4591-4600.
[12] SHUAI B，BERNESHAWI A G，MODOLO D，et al.Multi-object tracking with Siamese track-RCNN[J].arXiv：2004.07786，2020.
[13] RISTANI E，TOMASI C.Features for multi-target multi-camera tracking and re-identification[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：6036-6046.
[14] CHU P，LING H.FamNet：Joint learning of feature，affinity and multi-dimensional assignment for online multiple object tracking[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：6172-6181.
[15] VOIGTLAENDER P，KRAUSE M，OSEP A，et al.Mots：Multi-object tracking and segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：7942-7951.
[16] YANG L，FAN Y，XU N.Video instance segmentation[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：5188-5197.
[17] XU Z，ZHANG W，TAN X，et al.Segment as points for efficient online multi-object tracking and segmentation[C]//Proceedings of the European Conference on Computer Vision，2020：264-281.
[18] O?EP A，MEHNER W，VOIGTLAENDER P，et al.Track，then decide：Category-agnostic vision-based multi-object tracking[C]//Proceedings of the 2018 IEEE International Conference on Robotics and Automation（ICRA），2018：3494-3501.
[19] HE K，GKIOXARI G，DOLLáR P，et al.Mask R-CNN[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2961-2969.
[20] NEVEN D，BRABANDERE B D，PROESMANS M，et al.Instance segmentation by jointly optimizing spatial embeddings and clustering bandwidth[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：8837-8845.
[21] 刘晓栋，尚振宏，黄欢.有效块的多特征融合的目标跟踪方法[J].计算机工程与应用，2020，56（3）：201-206.
LIU X D，SHANG Z H，HUANG H.Target tracking method for multi-feature fusion of effective block[J].Computer Engineering and Applications，2020，56（3）：201-206.
[22] 王玲，王辉，王鹏，等.基于FasterMDNet的视频目标跟踪算法[J].计算机工程与应用，2020，56（14）：123-130.
WANG L，WANG H，WANG P，et al.Video target tracking algorithm using FasterMDNet[J].Computer Engineering and Applications，2020，56（14）：123-130.
[23] 陈云芳，吴懿，张伟.基于孪生网络结构的目标跟踪算法综述[J].计算机工程与应用，2020，56（6）：10-18.
CHEN Y F，WU Y，ZHANG W.Survey of target tracking algorithm based on Siamese network structure[J].Computer Engineering and Applications，2020，56（6）：10-18.
[24] LUITEN J，FISCHER T，LEIBE B.Track to reconstruct and reconstruct to track[J].IEEE Robotics and Automation Letters，2020，5（2）：1803-1810.
[25] RUIZ I，PORZI L，BULO S R，et al.Weakly supervised multi-object tracking and segmentation[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision，2021：125-133.
[26] PORZI L，HOFINGER M，RUIZ I，et al.Learning multi-object tracking and segmentation from automatic annotations[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：6846-6855.
[27] SONG Y，JEON M.Online multi-object tracking and segmentation with GMPHD filter and simple affinity fusion[J].arXiv：2009.00100，2020.
[28] LIN C C，HUNG Y，FERIS R，et al.Video instance segmentation tracking with a modified VAE architecture[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：13147-13157.
[29] YUAN Y，CHEN W，YANG Y，et al.In defense of the triplet loss again：Learning robust person re-identification with fast approximated triplet loss and label distillation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops，2020：354-355.
[30] KUHN H W.The Hungarian method for the assignment problem[J].Naval Research Logistics Quarterly，1955，2（1/2）：83-97.
[31] BERNARDIN K，STIEFELHAGEN R.Evaluating multiple object tracking performance：The clear mot metrics[J].EURASIP Journal on Image and Video Processing，2008，2008：1-10.
[32] OSEP A，MEHNER W，MATHIAS M，et al.Combined Proceedings of the image-and world-space tracking in traffic scenes[C]//Proceedings of 2017 IEEE International Conference on Robotics and Automation，2017：1988-1995.
[33] SHARMA S，ANSARI J A，MURTHY J K，et al.Beyond pixels：Leveraging geometry and shape cues for online multi-object tracking[C]//Proceedings of the 2018 IEEE International Conference on Robotics and Automation，2018：3508-3515.