Visual Tracking with Feature Aggregation Siamese Network Fused Attention

doi:10.3778/j.issn.1002-8331.2403-0273

Abstract

Abstract: At present, the target tracking algorithm based on siamese network still has the problem that the valuable context information in the shallow features of the network cannot be reasonably used. To solve this problem, a target tracking algorithm SiamMCFA (siamese multi-channel feature aggregation module) combining split-attention mechanism (SA) is proposed. Firstly, a split attention mechanism is introduced into the backbone network to extract valuable context information from the shallow features. Then, the context information in the shallow and deep features of the template region and the search region is fused by the pixel-wise cross correlation (PWCC) module. In order to enhance the connection between the feature maps of the template region and the search region, so as to improve the robustness of the tracker. Secondly, aiming at the problem of target loss caused by scale change, a multi-channel feature aggregation module (MCFA) is designed to aggregate the feature information of different regions of the target, so that the tracker can distinguish the target from the semantic background as much as possible and further improve the tracking accuracy. Finally, a detailed experimental evaluation is conducted on four datasets: OTB100, VOT2019, GOT10K, and LaSOT. The results show that compared with the current advanced tracker SiamCAR based on the siamese network, the success rate and precision of SiamMCFA have increased by 2.26 and 2.83 percentage points respectively. Compared with SiamIRCA, the success rate and accuracy have increased by 0.3 and 0.9 percentage points respectively.

Key words: object tracking, siamese network, split attention, pixel-wise cross correlation, multi-channel feature aggregation

摘要： 目前以孪生网络为基础的目标跟踪算法，仍然存在网络浅层的特征中有价值的上下文信息无法合理利用的问题。针对这一问题，提出一种融合拆分注意力机制（split-attention，SA）的目标跟踪算法SiamMCFA（siamese multi-channel feature aggregation module）。在骨干网络中引入拆分注意力机制，用来提取浅层特征中有价值的上下文信息，通过像素级互相关模块（pixel-wise cross correlation，PWCC）融合模板区域和搜索区域浅层和深层特征中的上下文信息，以增强模板区域和搜索区域的特征图之间的联系，从而提高跟踪器的鲁棒性。针对因尺度变化而容易导致目标丢失的问题，设计了一个多通道特征聚合模块（multi-channel feature aggregation module，MCFA），用于聚合目标不同区域的特征信息，使跟踪器尽可能地区分目标和语义背景，进一步提升跟踪准确性。最后，在OTB100、VOT2019、GOT10K和LaSOT四个数据集上进行了详尽的实验评估，结果显示，SiamMCFA与当前基于孪生网络的先进的跟踪器SiamCAR相比，其成功率（success rate）与精准度（precision）分别提高了2.26和2.83个百分点。与SiamIRCA相比成功率与精准度提高了0.3和0.9个百分点。

关键词: 目标跟踪, 孪生网络, 拆分注意力, 像素级互相关, 多通道特征聚合

JIN Jing, NIU Pin, ZHAI Fengwen. Visual Tracking with Feature Aggregation Siamese Network Fused Attention[J]. Computer Engineering and Applications, 2025, 61(12): 166-176.

金静, 牛品, 翟凤文. 融合注意力的特征聚合孪生网络视觉跟踪[J]. 计算机工程与应用, 2025, 61(12): 166-176.

References

[1] KUGARAJEEVAN J, KOKUL T, RAMANAN A, et al. Transformers in single object tracking: an experimental survey[J]. IEEE Access, 2023, 11: 80297-80326.
[2] TAO R, GAVVES E, SMEULDERS A W M. Siamese instance search for tracking[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 1420-1429.
[3] BERTINETTO L, VALMADRE J, HENRIQUES J F, et al. Fully-convolutional Siamese networks for object tracking[C]//Proceedings of the 2018 25th IEEE International Conference on Image Processing (ICIP). Cham: Springer International Publishing, 2018: 850-865.
[4] LI B, YAN J J, WU W, et al. High performance visual tracking with Siamese Region proposal network[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 8971-8980.
[5] LI B, WU W, WANG Q, et al. SiamRPN++: evolution of Siamese visual tracking with very deep networks[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 4277-4286.
[6] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 770-778.
[7] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84-90.
[8] GUO D Y, WANG J, CUI Y, et al. SiamCAR: siamese fully convolutional classification and regression for visual tracking[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 6268-6276.
[9] ZHANG Z P, PENG H W, FU J L, et al. Ocean: object-aware anchor-free tracking[C]//Proceedings of the 16th European Conference on Computer Vision (ECCV 2020). Cham: Springer International Publishing, 2020: 771-787.
[10] CHEN Z D, ZHONG B N, LI G R, et al. Siamese box adaptive network for visual tracking[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 6667-6676.
[11] CHAN S X, TAO J, ZHOU X L, et al. Siamese implicit region proposal network with compound attention for visual tracking[J]. IEEE Transactions on Image Processing, 2022, 31: 1882-1894.
[12] 王子康, 姚文进, 薛尚捷, 等. 基于深浅层特征融合的孪生网络目标跟踪算法[J]. 北京航空航天大学学报, 2025, 51(3): 973-984.
WANG Z K, YAO W J, XUE S J, et al. Siamese network object tracking algorithm based on deep and shallow feature fusion[J]. Journal of Beijing University of Aeronautics and Astronautics, 2025, 51(3): 973-984.
[13] LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2015: 3431-3440.
[14] WANG J Q, CHEN K, YANG S, et al. Region proposal by guided anchoring[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 2960-2969.
[15] DAI J F, LI Y, HE K M, et al. R-FCN: object detection via region-based fully convolutional networks[J]. arXiv:1605. 06409, 2016.
[16] LI B, YAN J J, WU W, et al. High performance visual tracking with siamese region proposal network[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 8971-8980.
[17] REVAUD J, WEINZAEPFEL P, HARCHAOUI Z, et al. DeepMatching: hierarchical deformable dense matching[J]. International Journal of Computer Vision, 2016, 120(3): 300-323.
[18] WANG Z, LIN J, LIU P. Ranking attention network for fast video object segmentation[C]//Proceedings of the IEEE International Conference on Computer Vision, 2019: 3044-3054.
[19] ZHANG H, WU C R, ZHANG Z Y, et al. ResNeSt: split-attention networks[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway: IEEE, 2022: 2735-2745.
[20] MA Y H, HAO H Y, XIE J Y, et al. ROSE: a retinal OCT-angiography vessel segmentation dataset and new model[J]. IEEE Transactions on Medical Imaging, 2021, 40(3): 928-939.
[21] FU J, LIU J, TIAN H J, et al. Dual attention network for scene segmentation[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 3141-3149.
[22] TIAN Z, SHEN C H, CHEN H, et al. FCOS: fully convolutional one-stage object detection[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2019: 9626-9635.
[23] RUSSAKOVSKY O, DENG J, SU H, et al. ImageNet large scale visual recognition challenge[J]. International Journal of Computer Vision, 2015, 115(3): 211-252.
[24] LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]//Proceedings of the 13th European Conference on Computer Vision. Cham: Springer International Publishing, 2014: 740-755.
[25] HUANG L H, ZHAO X, HUANG K Q. GOT-10k: a large high-diversity benchmark for generic object tracking in the wild[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(5): 1562-1577.
[26] FAN H, LIN L T, YANG F, et al. LaSOT: a high-quality benchmark for large-scale single object tracking[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 5369-5378.
[27] WU Y, LIM J, YANG M H. Object tracking benchmark[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1834-1848.
[28] KRISTAN M, MATAS J, LEONARDIS A, et al. The seventh visual object tracking VOT2019 challenge results[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshop. Piscataway: IEEE, 2019: 2206-2241.
[29] BHAT G, DANELLJAN M, VAN GOOL L, et al. Learning discriminative model prediction for tracking[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway: IEEE, 2019: 6181-6190.
[30] DANELLJAN M, BHAT G, KHAN F S, et al. ATOM: accurate tracking by overlap maximization[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 4655-4664.
[31] HU W M, WANG Q, ZHANG L, et al. SiamMask: a framework for fast online object tracking and segmentation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(3): 3072-3089.
[32] DONG X P, SHEN J B, PORIKLI F, et al. Adaptive siamese tracking with a compact latent network[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(7): 8049-8062.
[33] HUANG B, XU T F, SHEN Z Y, et al. SiamATL: online update of Siamese tracking network via attentional transfer learning[J]. IEEE Transactions on Cybernetics, 2022, 52(8): 7527-7540.
[34] 雷帮军, 丁奇帅, 牟乾西, 等. 基于模板更新和双特征增强的视觉跟踪算法[J/OL]. 北京航空航天大学学报, 2024: 1-15 (2024-02-27)[2024-03-05]. https://link.cnki.net/doi/10. 13700/j.bh. 1001-5965.2024.0020.
LEI B J, DING Q S, MOU Q X, et al. Visual tracking algorithm based on template updating and dual feature enhancement[J/OL]. Journal of Beijing University of Aeronautics and Astronautics, 2024: 1-15 (2024-02-27)[2024-03-05]. https://link.cnki.net/doi/10.13700/j.bh.1001-5965.2024.0020.
[35] 陈志旺, 杨天宇, 曹索航, 等. 基于稀疏注意力的孪生网络目标跟踪算法[J]. 控制与决策, 2024, 39(12): 4017-4026.
CHEN Z W, YANG T Y, CAO S H, et al. Siamese network object tracking algorithm based on sparse attention[J]. Control and Decision, 2024, 39(12): 4017-4026.
[36] DANELLJAN M, BHAT G, KHAN F S, et al. ECO: efficient convolution operators for tracking[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 6931-6939.
[37] TANG F, LING Q. Ranking-based siamese visual tracking[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2022: 8731-8740.
[38] DANELLJAN M, ROBINSON A, SHAHBAZ KHAN F, et al. Beyond correlation filters: learning continuous convolution operators for visual tracking[C]//Proceedings of the 2016 IEEE Conference on Computer Vision. Cham: Springer International Publishing, 2016: 472-488.
[39] WANG G T, LUO C, XIONG Z W, et al. SPM-tracker: series-parallel matching for real-time visual object tracking[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 3638-3647.
[40] BAO H, SHU P, ZHANG H C, et al. Siamese-based twin attention network for visual tracking[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(2): 847-860.
[41] 李俊, 曹林, 张帆, 等. 分布统计特征的孪生网络目标跟踪跟踪方法[J]. 计算机工程与应用, 2024, 60(8): 213-224.
LI J, CAO L, ZHANG F, et al. Siamese networks for object tracking on statistical characteristics of distributions[J]. Computer Engineering and Applications, 2024, 60(8): 213-224.
[42] WU Q Q, WAN J, CHAN A B. Progressive unsupervised learning for visual object tracking[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 2992-3001.