Research Progress of Multi-Target Tracking Based on Deep Learning from Perspective of UAV

doi:10.3778/j.issn.1002-8331.2307-0355

Abstract

Abstract: Multi-object tracking based on UAV platforms has wide application prospects in various fields, including smart cities, agricultural production, disaster early warning and search, and rescue operations. Unlike the relatively mature multi-object tracking from the traditional perspective, multi-object tracking from the UAV perspective faces a series of challenges that have not been completely solved. These challenges mainly include target scale changes, interference from similar targets, target occlusion and overlap, and uneven target distribution. This paper compiles the classic multi-object tracking algorithms developed in recent years from a traditional perspective. It also comprehensively analyzes the main technical approaches and latest methods in the field of multi-object tracking from the perspective of UAVs, with a particular focus on the detection-based tracking framework. Additionally, it examines the performance evaluation methods and mainstream datasets used in this domain. Moreover, the paper analyzes the primary challenges faced in multi-object tracking from UAV perspectives and offers insights into future research trends, aiming to provide valuable references for further related studies.

Key words: unmanned aerial vehicle（UAV）, multi-target tracking, deep learning, tracking by detection, joint detection tracking

摘要： 基于无人机平台的多目标跟踪在智慧城市、农业生产、灾害预警与搜救等多个领域有广泛的应用前景。与普通视角下的多目标跟踪相对成熟不同，无人机视角下的多目标跟踪面临一系列尚未完全解决的问题，主要包括目标尺度变化、相似目标干扰、目标被遮挡和重叠以及目标分布不均等。梳理了近年来在普通视角下的经典多目标跟踪算法，并以基于检测的跟踪框架为主，综合分析了无人机视角下多目标跟踪领域的主要技术路线和最新方法。统计了性能评估方法和主流数据集，分析了当前无人机视角下多目标跟踪所面临的主要挑战，并对未来的研究趋势进行展望，旨在为后续相关研究提供参考。

关键词: 无人机（UAV）, 多目标跟踪, 深度学习, 基于检测的跟踪, 联合检测的跟踪

YANG Yang, SONG Pinde, ZHONG Chunlai, CAO Lijia. Research Progress of Multi-Target Tracking Based on Deep Learning from Perspective of UAV[J]. Computer Engineering and Applications, 2023, 59(23): 48-62.

杨洋, 宋品德, 钟春来, 曹立佳. 无人机视角下基于深度学习的多目标跟踪研究进展[J]. 计算机工程与应用, 2023, 59(23): 48-62.

References

[1] JIMéNEZ-BRAVO D M，LOZANO M á，SALES MENDES A，et al.Multi-object tracking in traffic environments：a systematic literature review[J].Neurocomputing，2022，494：43-55.
[2] LUO W，XING J，MILAN A，et al.Multiple object tracking：a literature review[J].Artificial Intelligence，2021，293：103448.
[3] 刘艺，李蒙蒙，郑奇斌，等.视频目标跟踪算法综述[J].计算机科学与探索，2022，16（7）：1504-1515.
LIU Y，LI M M，ZHENG B Q，et al.Survey on video object tracking algorithms[J].Journal of Frontiers of Computer Science and Technology，2022，16（7）：1504-1515.
[4] 于丰华，张鸿刚，金忠煜，等.水稻农学理化参量无人机遥感反演研究现状与展望[J].沈阳农业大学学报，2023，54（2）：248-256.
YU F H，ZHANG H G，JIN Z Y，et al.Research status and prospect of unmanned aerial vehicle remote sensing in inversion of rice agronomic physicochemical parameters[J].Journal of Shenyang Agricultural University，2023，54（2）：248-256.
[5] 张换然，申凌峰，任资卓，等.无人机辅助智能边缘网络技术综述[J].电讯技术，2023（1）：1-10.
ZHANG H R，SHEN L F，REN Z Z，et al.A comprehensive survey on unmanned aerial vehicle assisted intelligent edge network[J].Telecommunication Engineering，2023（1）：1-10.
[6] XU Y，ZHOU X，CHEN S，et al.Deep learning for multiple object tracking：a survey[J].IET Computer Vision，2019，13（4）：355-368.
[7] PAL S K，PRAMANIK A，MAITI J，et al.Deep learning in multi-object detection and tracking：state of the art[J].Applied Intelligence，2021，51（9）：6400-6429.
[8] 陈琳，刘允刚.面向无人机的视觉目标跟踪算法：综述与展望[J].信息与控制，2022，51（1）：23-40.
CHEN L，LIU Y G.UAV visual target tracking algorithms：review and future prospect[J].Information and Control，2022，51（1）：23-40.
[9] 苑玉彬，吴一全，赵朗月，等.基于深度学习的无人机航拍视频多目标检测与跟踪研究进展[J].航空学报，2023（18）：1-31.
YUAN Y B，WU Y Q，ZHAO L Y，et al.Research progress of UAV aerial video multi-object detection and tracking based on deep learning[J].Acta Aeronautica et Astronautica Sinica，2023（18）：1-31.
[10] 葛宝义，左宪章，胡永江.视觉目标跟踪方法研究综述[J].中国图象图形学报，2018，23（8）：1091-1107.
GE B Y，ZUO X Z，HU Y J.Review of visual object tracking technology[J].Journal of Image and Graphics，2018，23（8）：1091-1107．
[11] BEWLEY A，GE Z，OTT L，et al.Simple online and realtime tracking[C]//Proceedings of the 2016 IEEE International Conference on Image Processing（ICIP），Phoenix，AZ，USA，2016：3464-3468.
[12] WOJKE N，BEWLEY A，PAULUS D.Simple online and realtime tracking with a deep association metric[C]//Proceedings of the 2017 IEEE International Conference on Image Processing（ICIP），Bejing，China，2017：3645-3649.
[13] CHEN L，AI H，ZHUANG Z，et al.Real-time multiple people tracking with deeply learned candidate selection and person re-identification[C]//Proceedings of the 2018 IEEE International Conference on Multimedia and Expo（ICME），Diego，CA，USA，2018：1-6.
[14] FU H，WU L，JIAN M，et al.MF-SORT：simple online and realtime tracking with motion features[C]//Proceedings of the 2019 International Conference on Image and Graphics（ICIG），Beijing，China，2019：157-168.
[15] AHARON N，ORFAIG R，BOBROVSKY B.BoT-SORT：robust associations multi-pedestrian tracking[J].arXiv：2206.14651，2022.
[16] ZHANG Y，SUN P，JIANG Y，et al.ByteTrack：multi-object tracking by associating every detection box[C]//Proceedings of the 2022 European Conference on Computer Vision（ECCV），Tel Aviv，Israel，2022：1-21.
[17] DU Y，ZHAO Z，SONG Y，et al.StrongSORT：make DeepSORT great again[J].arXiv：2202.13514，2022.
[18] CAO J，PANG J，WENG X，et al.Observation-centric SORT：rethinking SORT for robust multi-object tracking[C]//Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），Vancouver，Canada，2023：9686-9696.
[19] FEICHTENHOFER C，PINZ A，ZISSERMAN A.Detect to track and track to detect[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision （ICCV），Venice，Italy，2017：3038-3046.
[20] BERGMANN P，MEINHARDT T，LEAL-TAIXE L.Tracking without bells and whistles[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision（ICCV），Seoul，Korea （South），2019：941-951.
[21] ZHANG J，ZHOU S，CHANG X，et al.Multiple object tracking by flowing and fusing[J].arXiv：2001.11180，2020.
[22] ZHOU X，KOLTUN V，KR?HENBüHL P.Tracking objects as points[C]//Proceedings of the 2020 European Conference on Computer Vision（ECCV），Glasgow，UK，2020：474-490.
[23] ZHANG Y，WANG C，WANG X，et al.FairMOT：on the fairness of detection and re-identification in multiple object tracking[J].International Journal of Computer Vision，2021，129（11）：3069-3087.
[24] 潘昊，刘翔，赵静文，等.联合Transformer与BYTE数据关联的多目标实时跟踪算法[J].激光与光电子学进展，2023，60（6）：154-161.
PAN H，LIU X，ZHAO W J，et al.Multitarget real-time tracking algorithm based on Transformer and BYTE data[J].Laser & Optoelectronics Progress，2023，60（6）：154-161.
[25] 蒋凌云，杨金龙.检测优化的标签多伯努利视频多目标跟踪算法[J].计算机科学与探索，2021，17（6）：1343-1358.
JIANG L Y，YANG J L.Detection optimized labeled multi-bernoulli algorithm for visual multi-target tracking[J].Journal of Frontiers of Computer Science and Technology，2021，17（6）：1343-1358.
[26] 张英俊，白小辉，谢斌红.CNN-Transformer特征融合多目标跟踪算法[J].计算机工程与应用，2023：1-14（2023-04-11）.https：//link.cnki.net/urlid/11.2127.TP.20230411.1035.002.
ZHANG Y J，BAI X H，XIE B H.Multi-object tracking algorithm based on CNN-Transformer feature fusion[J].Computer Engineering and Applications，2023：1-14（2023-04-11）.https：//link.cnki.net/urlid/11.2127.TP.20230411.
1035.002.
[27] LI S，LIU Y，ZHAO Q，et al.Learning residue-aware correlation filters and refining scale for real-time UAV tracking[J].Pattern Recognition，2022，127：108614.
[28] ZHANG F，MA S，QIU Z，et al.Learning target-aware background-suppressed correlation filters with dual regression for real-time UAV tracking[J].Signal Processing，2022，191：108352.
[29] HE Y，FU C，LIN F，et al.Towards robust visual tracking for unmanned aerial vehicle with tri-attentional correlation filters[C]//Proceedings of the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems（IROS），Las Vegas，NV，USA，2020：1575-1582.
[30] WEN J，CHU H，LAI Z，et al.Enhanced robust spatial feature selection and correlation filter learning for UAV tracking[J].Neural Networks，2023，161：39-54.
[31] LI B，FU C，DING F，et al.ADTrack：target-aware dual filter learning for real-time anti-dark UAV tracking[C]//Proceedings of the 2021 IEEE International Conference on Robotics and Automation（ICRA），Xi’an，China，2021：496-502.
[32] MOHAMED B，HAJAR E K，NOURA J，et al.Strengthen aircraft real-time multiple object tracking with optical flow and histogram of oriented gradient provided HSMA implemented in low-cost energy VPU for UAV[C]//Proceedings of the 3rd International Conference on Innovative Research in Applied Science，Engineering and Technology（IRASET），Mohammedia，Morocco，2023：1-5.
[33] LIU Y，LI X，BAI T，et al.Multi-object tracking with hard-soft attention network and group-based cost minimization[J].Neurocomputing，2021，447：80-91.
[34] LUSARDI C，TAUFIQUE A M N，SAVAKIS A.Robust multi-object tracking using re-identification features and graph convolutional networks[C]//Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision（ICCV），Montreal，QC，Canada，2021：3868-3877.
[35] YU H，LI G，ZHANG W，et al.Self-balance motion and appearance model for multi-object tracking in UAV[C]//Proceedings of the MMAsia’19：ACM Multimedia Asia，Beijing，China，2019：1-6.
[36] BARTOLI A，FUSIELLO A.PAS Tracker：position-，appearance- and size-aware multi-object tracking in drone videos[C]//Proceedings of the 2021 16th European Conference on Computer Vision（ECCV），Glasgow，UK，2021：604-620.
[37] DIKE H U，ZHOU Y.A Robust quadruplet and faster region-based CNN for UAV video-based multiple object tracking in crowded environment[J].Electronics，2021，10（7）：795.
[38] 罗茜，赵睿，庄慧珊，等.YOLOv5与Deep-SORT联合优化的无人机多目标跟踪算法[J].信号处理，2022，38（12）：2628-2638.
LUO X，ZHAO R，ZHUANG H S，et al.UAV multi-target tracking algorithm jointly optimized by YOLOv5 and Deep-SORT[J].Journal of Signal Processing，2022，38（12）：2628-2638.
[39] MICHEAL A A，VANI K.Deep learning-based multi-class multiple object tracking in UAV video[J].Journal of the Indian Society of Remote Sensing，2022，50（12）：2543-2552.
[40] 孙晓迪，刘文楷，孙嘉怡，等.基于门限卷积孪生网络的无人机对地目标跟踪[C]//第十五届全国信号和智能信息处理与应用学术会议，中国重庆，2022：5.
SUN X D，LIU W K，SUN J Y，et al.UAV ground target tracking based on threshold convolutional siamese network[C]//Proceedings of the 15th National Conference on Signal and Intelligent Information Processing and Application，Chongqing，China，2022：5.
[41] ZHAO X，ZHANG Y.TFAtrack：temporal feature aggregation for UAV tracking and a unified benchmark[C]//Proceedings of the 2022 Chinese Conference on Pattern Recognition and Computer Vision（PRCV），Shenzhen，Guangdong，China，2022：55-66.
[42] XUE Y，JIN G，SHEN T，et al.Template-guided frequency attention and adaptive cross-entropy loss for UAV visual tracking[J].Chinese Journal of Aeronautics，2023（1）.
[43] XIONG L，TANG G.Multi-object tracking based on deep associated features for UAV applications[J].Optoelectronics Letters，2023，19（2）：105-111.
[44] ZHENG G，FU C，YE J，et al.Scale-aware siamese object tracking for vision-based UAM approaching[C]//Proceedings of the 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems，2022.
[45] WU H，HE Z，GAO M.GCEVT：learning global context embedding for vehicle tracking in unmanned aerial vehicle videos[J].IEEE Geoscience and Remote Sensing Letters，2022，20：1-5.
[46] CHEN T，PENNISI A，LI Z，et al.A hierarchical association framework for multi-object tracking in airborne videos[J].Remote Sensing，2018，10（9）：1347.
[47] LUO R，WEI J，LIN Q.VariabilityTrack：multi-object tracking with variable speed object movement[J].arXiv：2203.06424，2022.
[48] XIE W，IDE J，IZADI D，et al.Multi-object tracking with deep learning ensemble for unmanned aerial system applications[J].In Artificial Intelligence and Machine Learning in Defense Applications III，2021，11870：83-95.
[49] SAETCHNIKOV I，SKAKUN V，TCHERNIAVSKAIA E.Efficient objects tracking from an unmanned aerial vehicle[C]//Proceedings of the 2021 IEEE 8th International Workshop on Metrology for AeroSpace，Naples，Italy，2021：221-225.
[50] LIU S，LI X，LU H，et al.Multi-object tracking meets moving UAV[C]//Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），New Orleans，Louisiana，USA，2022：8876-8885.
[51] 苏昂，陆伟康，张仕林，等.基于目标运动模型的无人机对地视觉目标跟踪[J].激光与光电子学进展，2022，59（14）：243-251.
SU A，LU W K，ZHANG S L，et al.Visual ground target tracking of unmanned aerial vehicle based on target motion model[J].Laser & Optoelectronics Progress，2022，59（14）：243-251.
[52] XU X，FENG Z，CAO C，et al.STN-Track：multiobject tracking of unmanned aerial vehicles by swin transformer neck and new data association method[J].IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing，2022，15：8734-8743.
[53] 余仁伟，朱浩，蔡昌恺.基于薄板样条函数的无人机多目标跟踪算法[J].仪器仪表学报，2021，42（3）：168-176.
YU R W，ZHU H，CAI C K.Multi-object tracking algorithm for UAV based on the thin plate spline function[J].Chinese Journal of Scientific Instrument，2021，42（3）：168-176.
[54] CHENG S，YAO M，XIAO X.DC-MOT：motion deblurring and compensation for multi-object tracking in UAV videos[C]//Proceedings of the 2023 IEEE International Conference on Robotics and Automation（ICRA），London，UK，2023：789-795.
[55] ZHU J，YANG H，LIU N，et al.Online multi-object tracking with dual matching attention networks[C]//Proceedings of the 2018 European Conference on Computer Vision （ECCV），Munich，Germany，2018：366-382.
[56] PAN S，TONG Z，ZHAO Y，et al.Multi-object tracking hierarchically in visual data taken from drones[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision（ICCV），Seoul，Korea（South），2019：135-143.
[57] YU H，LI G，SU L，et al.Conditional GAN based individual and global motion fusion for multiple object tracking in UAV videos[J].Pattern Recognition Letters，2020，131：219-226.
[58] 牛畅，尹奎英，黄银和.无人机对地目标自动检测与跟踪技术[J].应用光学，2020，41（6）：1153-1160.
NIU C，YI K Y，HUANG Y H.Automatic target detecting and tracking technology based on UAV ground target images[J].Journal of Applied Optics，2020，41（6）：1153-1160.
[59] HUANG W，ZHOU X，DONG M，et al.Multiple objects tracking in the UAV system based on hierarchical deep high-resolution network[J].Multimedia Tools and Applications，2021，80（9）：13911-13929.
[60] WU H，NIE J，HE Z，et al.One-shot multiple object tracking in UAV videos using task-specific fine-grained features[J].Remote Sensing，2022，14（16）：3853.
[61] LIU Z，WANG S，YAO L，et al.Online multi-object tracking under moving unmanned aerial vehicle platform based on object detection and feature extraction network[J].Journal of Shanghai Jiaotong University（Science），2022.DOI：10.1007/s12204-022-2540-4.
[62] LIN Y，WANG M，CHEN W，et al.Multiple object tracking of drone videos by a temporal-association network with separated-tasks structure[J].Remote Sensing，2022，14（16）：3862.
[63] SUN P，CAO J，JIANG Y，et al.TransTrack：multiple object tracking with transformer[J].arXiv：2012.15460，2020.
[64] 武家鹏，韩煜祺，王旭辰，等.基于注意力机制的无人机对地多目标跟踪[C]//第十六届全国信号和智能信息处理与应用学术会议，2022：8.
WU J P，HAN Y Q，WANG X C，et al.UAV ground multi-object tracking based on attention mechanism[C]//Proceedings of the 16th National Conference on Signal and Intelligent Information Processing and Application，2022：8.
[65] SUN X，WANG Q，XIE F，et al.Siamese transformer network：building an autonomous real-time target tracking system for UAV[J].Journal of Systems Architecture，2022，130：102675.
[66] SHEN H，HUANG Z，WANG H，et al.Adaptive update of UAV multi-target tracking based on transformer[C]//Proceedings of the 2023 IEEE 6th Information Technology，Networking，Electronic and Automation Control Conference（ITNEC），Chongqing，China，2023：6-13.
[67] YE J，FU C，CAO Z，et al.Tracker meets night：a transformer enhancer for UAV tracking[J].IEEE Robotics and Automation Letters，2022，7（2）：3866-3873.
[68] BERNARDIN K，STIEFELHAGEN R.Evaluating multiple object tracking performance：the CLEAR MOT metrics[J].EURASIP Journal on Image and Video Processing，2008（1）：1-10.
[69] RISTANI E，SOLERA F，ZOU R，et al.Performance measures and a data set for multi-target，multi-camera tracking[C]//Proceedings of the 2016 European Conference on Computer Vision（ECCV），Amsterdam，Netherlands，2016：17-35.
[70] LUITEN J，OěEP A，DENDORFER P，et al.HOTA：a higher order metric for evaluating multi-object tracking[J].International Journal of Computer Vision，2021，129（2）：548-578.
[71] LEIBE B，MATAS J，SEBE N，et al.Learning social etiquette：human trajectory understanding in crowded scenes[C]//Proceedings of the 2016 European Conference on Computer Vision（ECCV），Amsterdam，Netherlands，2016：549-565.
[72] YU H，LI G，ZHANG W，et al.The unmanned aerial vehicle benchmark：object detection and tracking[C]//Proceedings of the 2018 European Conference on Computer Vision（ECCV），Munich，Germany，2018：370-386.
[73] ZHU P，WEN L，BIAN X，et al.Vision meets drones：a challenge[J].arXiv：1804.07437，2018.
[74] ZHU P，WEN L，DU D，et al.Vision meets drones：past，present and future[J].arXiv：2001.06303，2020.
[75] BONDI E，JAIN R，AGGRAWAL P，et al.BIRDSAI：a dataset for detection and tracking in aerial thermal infrared videos[C]//Proceedings of the 2020 IEEE/CVF Winter Conference on Applications of Computer Vision（WACV），Snowmass Village，CO，USA，2020：1736-1745.
[76] LI B，FU C，DING F，et al.All-day object tracking for unmanned aerial vehicle[J].IEEE Transactions on Mobile Computing，2022，22（8）：4515-4529.
[77] LI T，LIU J，ZHANG W，et al.UAV-human：a large benchmark for human behavior understanding with unmanned aerial vehicles[C]//Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition（CVPR），Nashville，TN，USA，2021：16266-16275.
[78] KAZANTSEV P A，SADAKOV V A，CHUPAKOV M V.Maritime vessels real-time tracking-by-detection in UAV videos[J].Indian Journal of Science and Technology，2017，9：1-9.
[79] XU S，SAVVARIS A，HE S，et al.Real-time implementation of YOLO+JPDA for small scale UAV multiple object tracking[C]//Proceedings of the 2018 International Conference on Unmanned Aircraft Systems（ICUAS），Dallas，TX，USA，2018：1336-1341.
[80] RATHORE A，SHARMA A，SHARMA N，et al.Multi-object tracking in heterogeneous environments（MOTHe） for animal video recordings[J].Peer J，2023，11：e15573.
[81] WANG S，JIANG F，ZHANG B，et al.Development of UAV-based target tracking and recognition systems[J].IEEE Transactions on Intelligent Transportation Systems，2020，21（8）：3409-3422.
[82] LIU X，ZHANG Z.A vision-based target detection，tracking，and positioning algorithm for unmanned aerial vehicle[J].Wireless Communications and Mobile Computing，2021（7）：1-12.
[83] LUO X，ZHAO R，GAO X.Research on UAV multi-object tracking based on deep learning[C]//Proceedings of the 2021 IEEE International Conference on Networking，Sensing and Control（ICNSC），Xiamen，Fujian，China，2021：1-6.
[84] CHEN Y，ZHAO D，ER M J，et al.A novel vehicle tracking and speed estimation with varying UAV altitude and video resolution[J].International Journal of Remote Sensing，2021，42（12）：4437-4462.