时空嵌入感知与多任务协同优化的多目标跟踪

doi:10.3778/j.issn.1002-8331.2211-0385

摘要/Abstract

摘要： 为解决多目标跟踪中遮挡频繁、场景拥挤以及目标尺度多变带来的跟踪挑战，提出时空嵌入感知与多任务协同优化的多目标跟踪方法。提出空间相关性模块以提取空间上带有目标上下文感知的判别力嵌入；提出时序相关性模块聚合来自空间相关性模块提取的嵌入，用于生成时序注意力以引导空间相关性模块在遮挡频繁和拥挤场景下提取更具判别力的嵌入。由此，判别力的嵌入在增强关联鲁棒性的同时可预测更加精确的检测框以克服尺度多变问题，而精确的检测框则促进两个模块提取更加高质量的嵌入，从而实现嵌入提取、位置预测和数据关联多任务间的协同优化。在亲和力矩阵中引入检测框间的GIoU距离以进一步提升遮挡和拥挤场景中关联的鲁棒性。在MOT16、MOT17和MOT20数据集上的实验结果表明，提出的方法表现出比先进方法更优异的跟踪性能。

关键词: 多目标跟踪, 时空嵌入感知, 位置预测, 数据关联, 协同优化

Abstract: To solve the tracking challenges caused by frequent occlusion, crowded scenes and variable object scales in multi-object tracking, a multi-object tracking method is proposed via spatial-temporal embedding perception and multi-task synergistic optimization. Firstly, spatial correlation module is proposed to extract discriminative embedding with object context awareness in spatial. Secondly, temporal correlation module is proposed to aggregate the embedding extracted from spatial correlation module, and aggregated embedding is used to generate temporal attention to guide spatial correlation module to extract more discriminative embedding in frequent occlusion and crowded scenes. Therefore, discriminative embedding enhances association robustness while predicting more accurate detection box to overcome the scale variability issues, and accurate detection box facilitates the extraction of higher quality embedding for the proposed modules. In this way, the synergistic optimization among multiple tasks of embedding extraction, position prediction and data association is achieved. Finally, GIoU distance among detection boxes is introduced into the affinity matrix to further improve association robustness in occlusion and crowded scenes. Experimental results on MOT16, MOT17 and MOT20 datasets show that the proposed method exhibits superior tracking performance to state-of-the-art methods.

Key words: multi-object tracking, spatial-temporal embedding perception, position prediction, data association, synergistic optimization

梁孝国, 李辉, 程远志, 陈双敏, 刘恒源. 时空嵌入感知与多任务协同优化的多目标跟踪[J]. 计算机工程与应用, 2024, 60(6): 282-292.

LIANG Xiaoguo, LI Hui, CHENG Yuanzhi, CHEN Shuangmin, LIU Hengyuan. Multi-Object Tracking with Spatial-Temporal Embedding Perception and Multi-Task Synergistic Optimization[J]. Computer Engineering and Applications, 2024, 60(6): 282-292.

参考文献

[1] 张瑶, 卢焕章, 张路平, 等. 基于深度学习的视觉多目标跟踪算法综述[J]. 计算机工程与应用, 2021, 57(13): 55-66.
ZHANG Y, LU H Z, ZHANG L P, et al. Overview of visual multi-object tracking algorithms with deep learning[J]. Computer Engineering and Applications, 2021, 57(13): 55-66.
[2] 方岚, 于凤芹. 自适应在线判别外观学习的分层关联多目标跟踪[J]. 中国图象图形学报, 2020, 25(4): 708-720.
FANG L, YU F Q. Multi-object tracking based on adaptive online discriminative appearance learning and hierarchical association[J]. Journal of Image and Graphics, 2020, 25(4): 708-720.
[3] BEWLEY A, GE Z Y, OTT L, et al. Simple online and realtime tracking[C]//Proceedings of the IEEE International Conference on Image Processing, Phoenix, September 25-28, 2016. Piscataway, NJ: IEEE Press, 2016: 3464-3468.
[4] 罗大鹏, 杜国庆, 曾志鹏, 等. 基于少量样本学习的多目标检测跟踪方法[J]. 电子学报, 2021, 49(1): 183-191.
LUO D P, DU G Q, ZENG Z P, et al. Multi-object detection and tracking based on few-shot learning[J]. Acta Electronica Sinica, 2021, 49(1): 183-191.
[5] YIN J B, WANG W G, MENG Q H, et al. A unified object motion and affinity model for online multi-object tracking[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Seattle, June 16-18, 2020. Piscataway, NJ: IEEE Press, 2020: 6767-6776.
[6] LIANG C, ZHANG Z P, ZHOU X, et al. Rethinking the competition between detection and ReID in multi-object tracking[J]. IEEE Transactions on Image Processing, 2022, 31(7): 3182-3196.
[7] SUN P, CAO J K, JIANG Y, et al. TransTrack: multiple object tracking with transformer[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Seattle, June 16-18, 2020. Piscataway, NJ: IEEE Press, 2020: 1-10.
[8] XU Y H, BAN Y T, DELORME G, et al. TransCenter: transformers with dense queries for multiple-object tracking[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Kuala Lumpur, June 19-25, 2021. Piscataway, NJ: IEEE Press, 2021:1-16.
[9] WANG G A, SONG M L, HWANG J N. Recent advances in embedding methods for multi-object tracking: a survey[EB/OL]. (2022-05-22)[2022-11-23]. https://arxiv.org/abs/2205.10766.
[10] MA C, YANG F, LI Y, et al. Deep human-interaction and association by graph-based learning for multiple object tracking in the wild[J]. International Journal of Computer Vision, 2021, 129(6): 1993-2010.
[11] 刘彦, 秦品乐, 曾建朝. 基于YOLOv3与分层数据关联的多目标跟踪算法[J]. 计算机科学, 2021, 48(S2): 370-375.
LIU Y, QIN P L, ZENG J C. Multi-object tracking algorithm based on YOLOv3 and hierarchical data association[J]. Computer Science, 2021, 48(S2): 370-375.
[12] 侯建华, 张国帅, 项俊. 基于深度学习的多目标跟踪关联模型设计[J]. 自动化学报, 2020, 46(12): 2690-2700.
HOU J H, ZHANG G S, XIANG J. Designing affinity model for multiple object tracking based on deep learning[J]. Acta Automatica Sinica, 2020, 46(12): 2690-2700.
[13] YU E, LI Z L, HAN S D, et al. RelationTrack: relation-aware multiple object tracking with decoupled representation[J/OL]. IEEE Transactions on Multimedia (2022-02-10)[2022-11-23].https://ieeexplore.ieee.org/document/9709649.
[14] BERGMANN P, MEINHARDT T, LEAL T L. Tracking without bells and whistles[C]//Proceedings of the IEEE International Conference on Computer Vision, Seoul, October 27-November 2, 2019. Piscataway, NJ: IEEE Press, 2019: 941-951.
[15] LIU Q K, CHU Q, LIU B, et al. GSM: graph similarity model for multi-object tracking[C]//Proceedings of the International Joint Conference on Artificial Intelligence, January 11-13, 2020. San Mateo: Morgan Kaufmann, 2020: 530-536.
[16] SADAT S F, SADEGH A M, SALZMANN M, et al. ArTIST: autoregressive trajectory inpainting and scoring for tracking [EB/OL]. (2020-04-16) [2022-11-23]. https://arxiv.org/abs/2004.07482.
[17] ZHOU X Y, KOLTUN V, KRAHENBUHL P. Tracking objects as points[C]//Proceedings of the IEEE European Conference on Computer Vision, August 23-28, 2020. Piscataway, NJ: IEEE Press, 2020: 474-490.
[18] REN S Q, HE K M, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 39(6): 1137-1149.
[19] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Las Vegas, June 26-July 1, 2016. Piscataway, NJ: IEEE Press, 2016: 770-778.
[20] LIN T Y, DOLLAR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Hawaii, July 21-26, 2017. Piscataway, NJ: IEEE Press, 2017: 936-944.
[21] LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]//Proceedings of the IEEE European Conference on Computer Vision, Zurich, September 5-12, 2014. Berlin: Springer, 2014: 740-755.
[22] BERNARDIN K, STIEFELHAGEN R. Evaluating multiple object tracking performance: the CLEAR MOT metrics[J]. EURASIP Journal on Image and Video Processing, 2008(1): 1-10.
[23] XU J R, CAO Y, ZHANG Z, et al. Spatial-temporal relation networks for multi-object tracking[C]//Proceedings of the IEEE International Conference on Computer Vision, Seoul, October 27-November 2, 2019. Piscataway, NJ: IEEE Press, 2019: 3987-3997.
[24] XU Y H, OSEP A, BAN Y T, et al. How to train your deep multi-object tracker[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Seattle, June 16-18, 2020. Piscataway, NJ: IEEE Press, 2020: 6786-6795.
[25] KIM C H, LI F X, MAZEN A, et al. Discriminative appearance modeling with multi-track pooling for real-time multi-object tracking[C]//Proceedings of the IEEE International Conference on Computer Vision, Montreal, October 11-18, 2021. Piscataway, NJ: IEEE Press, 2021: 9548-9557.
[26] GUO S, WANG J Y, WANG X C, et al. Online multiple object tracking with cross-task synergy[C]//Proceedings of the IEEE International Conference on Computer Vision, June 19-25, 2021. Piscataway, NJ: IEEE Press, 2021, 8132-8141.
[27] MA C, YANG F, LI Y, et al. Deep trajectory post-processing and position projection for single & multiple camera multiple object tracking[J]. International Journal of Computer Vision, 2021, 129(12): 3255-3278.
[28] YANG J M, GE H W, YANG J L, et al. Online multi-object tracking using multi-function integration and tracking simulation training[J]. Applied Intelligence, 2022, 52(2): 1268-1288.
[29] GAO T Z, PAN H H, WANG Z D, et al. A CRF-based framework for tracklet inactivation in online multi-object tracking[J]. IEEE Transactions on Multimedia, 2022, 24: 995-1007.
[30] SUN S J, AKHTAR N, SONG H S, et al. Deep affinity network for multiple object tracking[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(1): 104-119.
[31] WANG G A, WANG Y Z, GU R S, et al. Split and connect: a universal tracklet booster for multi-object tracking[J/OL]. IEEE Transactions on Multimedia (2022-01-06)[2022-11-23]. https://ieeexplore. ieee. org/document/9672670.
[32] PAPAKIS I, SARKAR A, KARPATNE A. GCNNMatch: graph convolutional neural networks for multi-object tracking via sinkhorn normalization[EB/OL]. (2021-04-16)[2022-11-23]. https://arxiv.org/abs/2010.00067.
[33] URBANN O, BREDTMANN O, OTTEN M, et al. Online and real-time tracking in a surveillance scenario[EB/OL]. (2021-06-02)[2022-11-23]. https://arxiv.org/abs/2106.01153.
[34] KARTHIK S, PRABHU A, GANDHI V. Simple unsupervised multi-object tracking[EB/OL]. (2020-06-04). [2022-11-23]. https://arxiv.org/abs/2006.02609.
[35] REN W H, WANG X C, TIAN J D, et al. Tracking-by-counting: using network flows on crowd density maps for tracking multiple targets[J]. IEEE Transactions on Image Processing, 2021, 30: 1439-1452.
[36] LI R, ZHANG B P, TENG Z, et al. An end-to-end identity association network based on geometry refinement for multi-object tracking[J]. Pattern Recognition, 2022, 129: 1-13.
[37] ZHANG Y, SHENG H, WU Y B, et al. Multiplex labeling graph for near-online tracking in crowded scenes[J]. IEEE Internet of Things Journal, 2020, 7(9): 7892-7902.