Detection and Tracking Method for Multiple Moving Objects in Panoramic Video

doi:10.3778/j.issn.1002-8331.2212-0371

Abstract

Abstract: The general solution of detection and tracking technology in panoramic videos is to detect and track moving objects after converting 360° video into two-dimensional video by projection. However, the fixed spherical projection will lead to discontinuous projection boundaries in panoramic videos. When a moving object is at a projection boundary, conventional spherical projection methods may divide the object into multiple parts, resulting in the missed or false detection. A multi-motion object detection and tracking method in panoramic video combining equi-rectangular projection (ERP) and cube map projection (CMP) is proposed. It uses suitable CMP stitching frames to reduce the polar distortion and compensate for the discontinuity of projection boundaries of ERP frames, solving the detection of objects at projection boundaries or polar distortion. Then, multi-motion object tracking is solved by fusing multiple features of perceptual hashing, spherical centroid and motion direction. Experimental results show that the proposed scheme can effectively alleviate the false detection caused by discontinuous projection boundaries and correctly track moving objects with occlusion, frame boundary shuttle or polar distortion.

Key words: panoramic video, multi-motion object detection and tracking, equi-rectangular projection (ERP), cube map projection (CMP)

摘要： 全景视频多运动对象检测与跟踪技术的通用解决方案是通过投影变换将360°视频转为二维视频后进行检测和跟踪。然而固定球面投影变换会导致全景视频中存在空间上不连续的投影边界，当运动对象处于投影边界时，常规球面投影方法可能将其切分为多个部分，从而导致该运动对象的漏检或误检。针对该问题，提出了融合等距柱状形投影（equi-rectangular projection，ERP）和立方体投影（cube map projection，CMP）的全景视频多运动对象检测与跟踪方法，利用合适的CMP拼接帧减少ERP两极区域失真并弥补其边界不连续性，解决了处于投影边界和两极扭曲的对象检测问题。进一步，通过感知哈希、球面质心和运动方向等多重特征融合来解决全景视频多运动对象跟踪问题。实验结果表明，提出的方案可以有效减少投影边界不连续导致的误检，且能有效应对遮挡、穿梭帧边界和两极扭曲的对象跟踪问题。

关键词: 全景视频, 多运动对象检测与跟踪, 等距柱状形投影（ERP）, 立方体投影（CMP）

LIU Huitong, WANG Xi, LIU Wei, YANG Peng. Detection and Tracking Method for Multiple Moving Objects in Panoramic Video[J]. Computer Engineering and Applications, 2024, 60(10): 188-197.

刘慧彤, 王希, 刘威, 杨鹏. 全景视频中多运动对象检测与跟踪方法[J]. 计算机工程与应用, 2024, 60(10): 188-197.

References

[1] 宋聪聪, 高策, 张艳超，等.基于三维球面模型的全景视频实时拼接方法[J].光学学报, 2023, 43(10): 73-86.
SONG C C, GAO C, ZHANG Y C, et al. Real-time stitching method of panoramic video based on three-dimensional spherical model[J]. Acta Optica Sinica, 2023, 43(10): 73-86.
[2] 王殿伟, 赵梦影, 刘颖, 等. 改进的R-SSD全景视频图像车辆检测算法[J].计算机工程与应用, 2021, 57(3): 189-195.
WANG D W, ZHAO M Y, LIU Y, et al. Improved R-SSD panoramic video image vehicle detection algorithm[J]. Computer Engineering and Applications, 2021, 57(3): 189-195.
[3] CHOPRA L, CHAKRABORTY S, MONDAL A, et al. PARIMA: viewport adaptive 360-degree video streaming[C]//Proceedings of the Web Conference, 2021: 2379-2391.
[4] FINNIE S, ZHANG F L, RHEE T. Visual object tracking in spherical 360° videos: a bridging approach[C]//Proceedings of the 2020 35th International Conference on Image and Vision Computing, New Zealand, 2020: 1-6.
[5] YANG W, QIAN Y, K?M?R?INEN J K, et al. Object detection in equirectangular panorama[C]//Proceedings of the 2018 24th International Conference on Pattern Recognition, 2018: 2190-2195.
[6] HE Y, XIU X, HANHART P, et al. Content-adaptive 360-degree video coding using hybrid cubemap projection[C]//Proceedings of the 2018 Picture Coding Symposium, 2018: 313-317.
[7] MI T W, YANG M T. Comparison of tracking techniques on 360-degree videos[J]. Applied Sciences, 2019, 9(16): 3336.
[8] 王殿伟, 方浩宇, 刘颖, 等. 一种基于改进RT-MDNet的全景视频目标跟踪算法[J]. 哈尔滨工业大学学报, 2020, 52(10): 152-160.
WANG D W, FANG H Y, LIU Y, et al. Improved RT-MDNet for panoramic video target tracking[J]. Journal of Harbin Institute of Technology, 2020, 52(10): 152-160.
[9] LIU K C, SHEN Y T, CHEN L G. Simple online and realtime tracking with spherical panoramic camera[C]//Proceedings of the 2018 IEEE International Conference on Consumer Electronics, 2018: 1-6.
[10] COORS B, CONDURACHE A P, GEIGER A. SphereNet: learning spherical representations for detection and classification in omnidirectional images[C]//Proceedings of the 15th European Conference on Computer Vision, 2018: 518-533.
[11] JIA P, TIE Y, QI L, et al. PV-YOLO: an object detection model for panoramic video based on YOLOv4[C]//Proceedings of the 2022 Asia Conference on Algorithms, Computing and Machine Learning, 2022: 56-61.
[12] CAO M, IKEHATA S, AIZAWA K. Dual-ERP representation for object detection in 360° images[C]//Proceedings of the 2022 IEEE International Conference on Image Processing, 2022: 2016-2020.
[13] LEE Y, JEONG J, YUN J, et al. SpherePHD: applying CNNs on a spherical polyhedrons representation of 360° images[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019: 9181-9189.
[14] LI M, MENG M, ZHOU Z. RepF-Net: distortion-aware re-projection fusion network for object detection in panorama image[C]//Proceedings of the 2022 Asian Conference on Computer Vision, 2022: 74-89.
[15] 黄天果, 何嘉, 沈庆阳. 基于全景视觉图像的实时目标检测方法[J]. 计算机工程与设计, 2022, 43(7): 2055-2061.
HUANG T G, HE J, SHEN Q Y. Real time object detection method based on panoramic image[J]. Computer Engineering and Design, 2022, 43(7): 2055-2061.
[16] 赵元龙, 单玉刚, 袁杰.改进YOLOv7与DeepSORT的佩戴口罩行人跟踪[J].计算机工程与应用, 2023, 59(6): 221-230.
ZHAO Y L, DAN Y G, YUAN J. Wearing mask pedestrian tracking based on improved YOLOv7 and DeepSORT[J]. Computer Engineering and Applications, 2023, 59(6): 221-230.
[17] DU L, HO A T S, CONG R. Perceptual hashing for image authentication: a survey[J]. Signal Processing: Image Communication, 2020, 81: 115713.
[18] WANG C Y, BOCHKOVSKIY A, LIAO H Y M. YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[J] arXiv:2207.02696, 2022.
[19] WU C, TAN Z, WANG Z, et al. A dataset for exploring user behaviors in VR spherical video streaming[C]//Proceedings of the 8th ACM Multimedia Systems Conference, 2017: 193-198.