Review on Detection of Drones and Birds in Photoelectric Images Based on Deep Learning Convolutional Neural Network

doi:10.3778/j.issn.1002-8331.2305-0372

Abstract

Abstract: With the development of the civilian drone industry, drones have become a critical issue affecting public safety. At present, the surveillance method for low-altitude drones mainly adopts the method of radar detection combined with visible image identification. However, visible image recognition is susceptible to interference from flying birds, which belongs to the same “low, slow, and small” targets as UAVs. To eliminate the interference of flying bird targets in the detection of UAVs based on visible images, the deep neural network is used to accurately identify and classify UAVs and flying birds in visible images, and effectively eliminate the interference of birds in the detection of UAVs. This paper first systematically explains the development process of target detection technology, discusses the differences of various target detection algorithms based on deep learning network, and compares the advantages and disadvantages of various algorithms. The image data sets that can be used for drone and bird detection are sorted out and introduced, and the existing results of related research are analyzed. Then, starting from the practical application, the problems that may exist in the detection of drones and birds are sorted out, and the research on neural networks that can solve the corresponding detection problems is elaborated and analyzed. In the end, the probable future directions of this research are prospected.

Key words: deep learning, convolutional neural network, target detection, drone, flying bird detection

摘要： 随着民用无人机产业的发展，无人机已经成为一项影响公共安全的重要问题。目前针对低空无人机的监视手段主要是采用雷达探测结合图像识别的方法，然而在图像检测识别中易受到与无人机同属“低、慢、小”目标的飞鸟的干扰。为了能够在基于可见光图像的无人机检测中排除飞鸟目标的干扰，利用深度神经网络对可见光图像中无人机与飞鸟进行精确的检测与分类，有效地排除飞鸟对无人机检测的干扰。系统阐释了目标检测技术的发展历程，讨论了各类基于深度学习网络目标检测算法的差异，对比了各类算法的优缺点。对可用于无人机与飞鸟检测的图像数据集进行了梳理与介绍，对相关研究的已有成果进行分析；再从实际应用出发，对无人机与飞鸟检测当中可能会存在的问题进行梳理，阐述与分析了能解决相应检测问题的卷积神经网络的相关研究。最后，针对该研究后续可能的发展方向进行展望。

关键词: 深度学习, 卷积神经网络, 目标检测, 无人机, 飞鸟检测

XIE Weiyu, ZHANG Qiang. Review on Detection of Drones and Birds in Photoelectric Images Based on Deep Learning Convolutional Neural Network[J]. Computer Engineering and Applications, 2024, 60(8): 46-55.

谢威宇, 张强. 基于深度学习的图像中无人机与飞鸟检测研究综述[J]. 计算机工程与应用, 2024, 60(8): 46-55.

References

[1] 罗俊海, 王芝燕. 无人机探测与对抗技术发展及应用综述[J].控制与决策, 2022, 37(3): 530-544.
LUO J H, WANG Z Y. A review of development and application of UAV detection and counter technology[J].Control and Decision, 2022, 37(3): 530-544.
[2] 陈唯实, 黄毅峰, 陈小龙, 等. 机场净空区飞鸟与非合作无人机目标识别[J].民航学报, 2020, 4(3): 27-33.
CHEN W S, HUANG Y F, CHEN X L, et al. Recognition methods of flying bird and non-cooperative drone targets in airport clearance aera[J].Journal of Civil Aviation, 2020, 4(3): 27-33.
[3] 张灵灵. 基于改进YOLOv5的低空无人机检测方法研究[D]. 西安：西安工业大学, 2023.
ZHANG L L. Research on low-altitude drone detection method based on improved YOLOv5[D]. Xi’an：Xi’an Technological University, 2023.
[4] 梅枫, 高兴宇, 邓仕超, 等. 双目测距和YOLOv5s的无人机快速识别定位追踪系统[J].现代电子技术, 2023, 46(10): 181-186.
MEI F, GAO X Y, DENG S C, et al. UAV rapid identification, positioning and tracking system based on binocular ranging and YOLOv5s[J].Modern Electronics Technique, 2023, 46(10): 181-186.
[5] VIOLA P, JONES M. Rapid object detection using a boosted cascade of simple features[C]//Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001: 511-518.
[6] VIOLA P, JONES M J. Robust real-time face detection[J].International Journal of Computer Vision, 2004, 57(2): 137-154.
[7] 卢宏涛, 张秦川. 深度卷积神经网络在计算机视觉中的应用研究综述[J].数据采集与处理, 2016, 31(1): 1-17.
LU H T, ZHANG Q C. Applications of deep convolutional neural network in computer vision[J].Journal of Data Acquisition and Processing, 2016, 31(1): 1-17.
[8] LECUN Y, et al. Handwritten digit recognition with a back-propagation network[C]//Advances in Neural Information Processing Systems 2, 1989: 396-404.
[9] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. Imagenet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems 25, 2012: 1097-1105.
[10] GIRSHICK R, DONAHUE J, DARRELL T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014: 580-587.
[11] HE K, ZHANG X, REN S, et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916.
[12] GIRSHICK R. Fast R-CNN[C]//Proceedings of the IEEE International Conference on Computer Vision, 2015: 1440-1448.
[13] REN S, HE K, GIRSHICK R, et al. Faster R-CNN: towards real?time object detection with region proposal networks[C]//Advances in Neural Information Processing Systems 28, 2015: 91-99.
[14] LIN T Y, DOLLáR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017: 2117-2125.
[15] REDMON J, DIVVALA S, GIRSICK R, et al. You only look once: unified, real-time object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 779-788.
[16] LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot multibox detector[C]//Proceedings of the 14th European Conference on Computer Vision, 2016: 21-37.
[17] LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(2): 318-327.
[18] DAI J, LI Y, HE K, et al. R-FCN: object detection via region-based fully convolutional networks[C]//Advances in Neural Information Processing Systems 29, 2016: 379-387.
[19] HE K, GKIOXARI G, DOLLáR P, et al. Mask R-CNN[C]//Proceedings of the IEEE International Conference on Computer Vision, 2017: 2961-2969.
[20] REDMON J, FARHADI A. YOLO9000: better, faster, stronger[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 7263-7271.
[21] REDMON J, FARHADI A. YOLOv3: an incremental improvement[J]. arXiv:1804.02767, 2018.
[22] CARION N, MASSA F, SYNNAEVE G, et al. End-to-end object detection with transformers[C]//Proceedings of the 16th European Conference on Computer Vision, 2020: 213-229.
[23] BOCHKOVSKIY A, WANG C Y, LIAO H Y M. YOLOv4: optimal speed and accuracy of object detection[J].arXiv:2004.10934, 2020.
[24] WANG C Y, BOCHKOVSKIY A, LIAO H Y M. YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 7464-7475.
[25] LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]//Proceedings of the 13th European Conference on Computer Vision, 2014: 740-755.
[26] EVERINGHAM M, ESLAMI S M A, VAN G L, et al. The pascal visual object classes challenge: a retrospective[J].International Journal of Computer Vision, 2015, 111(1): 98-136.
[27] WAH C, BRANSON S, WELINDER P, et al. The caltech-ucsd birds-200-2011 dataset[D].California Institute of Technology, 2011.
[28] LI J, YE D H, CHUNG T, et al. Multi-target detection and tracking from a single camera in unmanned aerial vehicles (UAVs)[C]//Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016: 4992-4997.
[29] ROZANTSEV A, LEPETIT V, FUA P. Detecting flying objects using a single moving camera[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 39(5): 879-892.
[30] CHEN Y, AGGARWAL P, CHOI J, et al. A deep learning approach to drone monitoring[C]//Proceedings of the Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, 2017: 686-691.
[31] COLUCCIA A, FASCISTA A, SCHUMANN A, et al. Drone vs. bird detection: deep learning algorithms and results from a grand challenge[J].Sensors, 2021, 21(8): 2824.
[32] SVANSTR?M F, ENGLUND C, ALONSO F F. Real-time drone detection and tracking with visible, thermal and acoustic sensors[C]//Proceedings of the 25th International Conference on Pattern Recognition (ICPR), 2021: 7265-7272.
[33] PAWE?CZYK M, WOJTYRA M. Real world object detection dataset for quadcopter unmanned aerial vehicle detection[J].IEEE Access, 2020, 8: 174394-174409.
[34] WALTER V, VRBA M, SASKA M. On training datasets for machine learning-based visual relative localization of micro-scale UAVs[C]//Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2020: 10674-10680.
[35] ZHENG Y, CHEN Z, LV D, et al. Air-to-air visual detection of micro-uavs: an experimental evaluation of deep learning[J].IEEE Robotics and Automation Letters, 2021, 6(2): 1020-1027.
[36] JIANG N, WANG K, PENG X, et al. Anti-UAV: a large multi-modal benchmark for UAV tracking[J]. arXiv:2101.08466, 2021.
[37] 蒋兆军, 成孝刚, 彭雅琴, 等. 基于深度学习的无人机识别算法研究[J].电子技术应用, 2017, 43(7): 84-87.
JIANG Z J, CHENG X G, PENG Y Q, et al. A novel UAV recognition algorithm based on deep learning approach[J].Application of Electronic Technique, 2017, 43(7): 84-87.
[38] COLUCCIA A, GHENESCU M, PIATRIK T, et al. Drone-vs-Bird detection challenge at IEEE AVSS2017[C]//Proceedings of the 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2017.
[39] SCHUMANN A, SOMMER L, KLATTE J, et al. Deep cross-domain flying object classification for robust UAV detection[C]//Proceedings of the 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2017.
[40] COLUCCIA A, SAQIB M, SHARMA N, et al. Drone-vs-Bird detection challenge at IEEE AVSS2019[C]//Proceedings of the 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2019.
[41] FERNANDES A, BAPTISTA M, FERNANDES L, et al. Drone, aircraft and bird identification in video images using object tracking and residual neural networks[C]//Proceedings of the 11th International Conference on Electronics, Computers and Artificial Intelligence (ECAI), 2019.
[42] MEDIAVILLA C, NANS L, MAREZ D, et al. Detecting aerial objects: drones, birds, and helicopters[J].Artificial Intelligence and Machine Learning in Defense Applications III, 2021, 11870: 83-97.
[43] ASHRAF M W, SULTANI W, SHAH M. Dogfight: detecting drones from drones videos[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 7067-7076.
[44] AKYON F C, AKAGUNDUZ E, ALTINUC S O, et al. Sequence models for drone vs. bird classification[J].arXiv:2207.10409, 2022.
[45] IHEKORONYE V U, AJAKWE S O, KIM D S, et al. Aerial supervision of drones and other flying objects using convolutional neural networks[C]//Proceedings of the International Conference on Artificial Intelligence in Information and Communication (ICAIIC), 2022: 69-74.
[46] LIU S, LI G, ZHAN Y, et al. MUSAK: a multi-scale space kinematic method for drone detection[J].Remote Sensing, 2022, 14(6): 1434.
[47] SOMMER L, SCHUMANN A, M?LLER T, et al. Flying object detection for automatic UAV recognition[C]//Proceedings of the 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2017.
[48] CAI Z, VASCONCELOS N. Cascade R-CNN: delving into high quality object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018: 6154-6162.
[49] XIE S, GIRSHICK R, DOLLáR P, et al. Aggregated residual transformations for deep neural networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 1492-1500.
[50] WANG X, YU K, WU S, et al. Esrgan: enhanced super-resolution generative adversarial networks[C]//Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018.
[51] KIM J H, KIM N, WON C S. High-speed drone detection based on YOLOv8 Detecting aerial objects: drones, birds, and helicopters[C]//Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
[52] MISTRY S K, CHATTERJEE S, VERMA A K, et al. Drone-vs-Bird: drone detection using YOLOv7 with CSRT tracker detecting aerial objects: drones, birds, and helicopters[C]//Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023.
[53] YU F, KOLTUN V. Multi-scale context aggregation by dilated convolutions[J].arXiv:1511.07122, 2015.
[54] CRAYE C, ARDJOUNE S. Spatio-temporal semantic segmentation for drone detection[C]//Proceedings of the 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2019.
[55] MAGOULIANITIS V, ATALOGLOU D, DIMOU A, et al. Does deep super-resolution enhance uav detection?[C]//Proceedings of the 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS), 2019.
[56] XU N, PRICE B, COHEN S, et al. Deep image matting[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 2970-2979.
[57] AFIFI M, DERPANIS K G, OMMER B, et al. Learning multi-scale photo exposure correction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 9157-9167.
[58] CHEN Z, WANG Y, YANG Y, et al. PSD: principled synthetic-to-real dehazing guided by physical priors[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 7180-7189.
[59] DONG H, PAN J, XIANG L, et al. Multi-scale boosted dehazing network with dense feature fusion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 2157-2167.
[60] LI L, DONG Y, REN W, et al. Semi-supervised image dehazing[J].IEEE Transactions on Image Processing, 2019, 29: 2766-2779.
[61] SHAO Y, LI L, REN W, et al. Domain adaptation for image dehazing[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020: 2808-2817.
[62] LI B, REN W, FU D, et al. Benchmarking single-image dehazing and beyond[J].IEEE Transactions on Image Processing, 2018, 28(1): 492-505.
[63] ZHONG Z, ZHENG Y, SATO I. Towards rolling shutter correction and deblurring in dynamic scenes[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 9219-9228.
[64] CHEN L, ZHANG J, PAN J, et al. Learning a non-blind deblurring network for night blurry images[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 10542-10550.
[65] LIU Q, LU X, HE Z, et al. Deep convolutional neural networks for thermal infrared object tracking[J].Knowledge-Based Systems, 2017, 134: 189-198.
[66] ZHOU X, YIN T, KOLTUN V, et al. Global tracking transformers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 8771-8780.
[67] TAO A, SAPRA K, CATANZARO B. Hierarchical multi-scale attention for semantic segmentation[J].arXiv:2005.10821, 2020.