Review of Object Detection Based on Event Cameras

doi:10.3778/j.issn.1002-8331.2312-0322

Abstract

Abstract: Event cameras are imaging methods that mimic biological retinas, with high dynamics, low latency, high temporal resolution and low power consumption. It breaks through the dilemma that traditional cameras are difficult to capture objects and target recognition under high dynamic range, and the characteristics of event cameras are of experimental significance for studying the object detection problem based on event cameras. This paper first briefly describes the status, development process, advantages and challenges of event cameras, then introduces the working principle of various types of event cameras and some object detection algorithms based on event cameras, and finally explains the challenges and future trends of object detection algorithms based on event cameras, and summarizes the article.

Key words: event cameras, object detection, neural networks

摘要： 事件相机是模仿生物视网膜的成像方式，具有高动态、低延迟、高时间分辨率以及低功耗的特性。其突破传统相机难以捕捉在高动态范围情况下的物体并进行目标识别的困境，事件相机的特性对于研究基于事件相机的目标检测问题具有实验意义。简要叙述事件相机的现状、发展过程、优势与挑战，介绍了各种类型事件相机的工作原理和一些基于事件相机的目标检测算法，阐述了基于事件相机的目标检测算法面对的挑战和未来趋势，并进行了总结。

关键词: 事件相机, 目标检测, 神经网络

ZHANG Yali, TIAN Qichuan, TANG Chaolin. Review of Object Detection Based on Event Cameras[J]. Computer Engineering and Applications, 2024, 60(13): 23-35.

张亚丽, 田启川, 唐超林. 基于事件相机的目标检测算法研究[J]. 计算机工程与应用, 2024, 60(13): 23-35.

References

[1] FUKUSHIMA K, YAMAGUCHI Y, YASUDA M, et al. An electronic model of the retina[J]. Proceedings of the IEEE, 1970, 58(12): 1950-1951.
[2] LICHTSTEINER P, POSCH C, DELBRUCK T. A 128×128 120 dB 15 μs latency asynchronous temporal contrast vision sensor[J]. IEEE Journal of Solid-State Circuits, 2008, 43(2): 566-576.
[3] DELBRUCK T, LICHTSTEINER P. Fast sensory motor control based on event-based hybrid neuromorphic-procedural system[C]//Proceedings of the IEEE International Symposium on Circuits and Systems, New Orleans, LA, USA, 2007: 845-848.
[4] CONRADT J, BERNER R, COOK M, T, et al. An embedded AER dynamic vision sensor for low-latency pole balancing[C]//Proceedings of the IEEE 12th International Conference on Computer Vision Workshops, Kyoto, Japan, 2009: 780-785.
[5] CONRADT J, COOK M, BERNER R, et al. A pencil balancing robot using a pair of AER dynamic vision sensors[C]//Proceedings of the IEEE International Symposium on Circuits and Systems, Taipei, Taiwan, China, 2009: 781-784.
[6] LITZENBERGER M, KOHN B, BELBACHIR A N, et al. Estimation of vehicle speed based on asynchronous data from a silicon retina optical sensor[C]//Proceedings of the IEEE Intelligent Transportation Systems Conference, Toronto, ON, Canada, 2006: 653-658.
[7] SCHRAML S, BELBACHIR A. N, MILOSEVIC N, et al. Dynamic stereo vision system for real-time tracking[C]//Proceedings of 2010 IEEE International Symposium on Circuits and Systems, Paris, France, 2010: 1409-1412.
[8] DRAZEN D, LICHTSTEINER P, HAFLIGER P, et al. Toward real-time particle tracking using an event-based dynamic vision sensor[J]. Experiments in Fluids, 2011, 51(55): 1465-1469.
[9] NI Z, PACORET C, BENOSMAN R B, et al. Asynchronous event‐based high speed vision for microparticle tracking[J]. Microscopy, 2012, 245(3): 236-244.
[10] LEE J, DELBRUCK T, PARK P, et al. Live demonstration: gesture based remote control using stereo pair of dynamic vision sensors[C]//Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS), Seoul, Korea (South), 2012: 741-745.
[11] POSCH C, MATOLIN D, WOHLGENANNT R. A QVGA 143 dB dynamic range frame-free PWM image sensor with lossless pixel-level video compression and time-domain CDS[J]. IEEE Journal of Solid-State Circuits, 2010, 46(1): 259-275.
[12] BRANDLI C, BERNER R, YANG M, et al. A 240×180 130 dB 3 μs latency global shutter spatiotemporal vision sensor[J]. Solid-State Circuits, 2014, 49(10): 2333-2341.
[13] MOEYS D P, CORRADI F, LI C H, et al. A sensitive dynamic and active pixel vision sensor for color or neural imaging applications[J]. IEEE Transactions on Biomedical Circuits and Systems, 2018, 12(1): 123-136.
[14] AMIR A, TABA B, BERG D, et al. A low power, fully event-based gesture recognition system[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017: 7388-7397.
[15] SERRANO-GOTARREDONA R, OSTER M, LICHTSTEINER P, et al. CAVIAR: a 45k neuron, 5M synapse, 12G connects/s AER hardware sensory-processing-learning-actuating system for high-speed visual object recognition and tracking[J]. IEEE Transactions on Neural Networks, 2009, 20(9): 1417-1438.
[16] XU H, GAO Y, YU F, et al. End-to-end learning of driving models from large-scale video datasets[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017: 3530-3538.
[17] LEVINSHTEIN A, STERE A, KUTULAKOS K N, et al. Fast super-pixels using geometric flows[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(12): 2290-2297.
[18] BOCHKOVSKIY A, WANG C Y, LIAO H Y M. YOLOv4: optimal speed and accuracy of object detection[J]. arXiv:2004.10934, 2020.
[19] WIESMANN G, SCHRAML S, LITZENBERGER M, et al. Event-driven embodied system for feature extraction and object recognition in robotic applications[C]//Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Providence, RI, USA, 2012: 76-82.
[20] CLADY X, IENG S H, BENOSMAN R. Asynchronous event-based corner detection and matching[J]. Neural Networks, 2015, 66: 91-106.
[21] VALENTINA V, GLOVER A, BARTOLOZZI C. Fast event-based Harris corner detection exploiting the advantages of event-driven cameras[C]//Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, Korea (South), 2016: 4144-4149.
[22] MUEGGLER E, REBECQ H, GALLEGO G, et al. The event-camera dataset and simulator: event-based data for pose estimation, visual odometry, and SLAM[J]. The International Journal of Robotics Research, 2017, 36(2): 142-149.
[23] PENG X, ZHAO B, YAN R, et al. Bag of events: an efficient probability-based feature extraction method for AER image sensors[J]. IEEE Transactions on Neural Networks and Learning Systems, 2017, 28(4): 791-803.
[24] RAMESH B, YANG H, ORCHARD G, et al. DART: distribution aware retinal transform for event-based cameras[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(11): 2767-2780.
[25] AFSHAR S, COHEN G, WANG R, et al. The ripple pond: enabling spiking networks to see[J]. Frontiers in Neuroscience, 2013, 7: 212.
[26] YAO M, GAO H H, ZHAO G S, et al. Temporal-wise attention spiking neural networks for event streams classification[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 2021: 10201-10210.
[27] NAGARAJ M, LIYANAGEDERA C M, ROY K. DOTIE - detecting objects through temporal isolation of events using a spiking architecture[C]//Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, United Kingdom, 2023: 4858-4864.
[28] ZHAO C H, LI Y, LYU Y. Event-based real-time moving object detection based on IMU ego-motion compensation[C]//Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, United Kingdom, 2023: 690-696.
[29] ZHU A Z, YUAN L, CHANEY K, et al. Unsupervised event-based learning of optical flow, depth, and egomotion[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019: 989-997.
[30] WANG Y, DU B, SHEN Y, et al. EV-gait: event-based robust gait recognition using dynamic vision sensors[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019: 6351-6360.
[31] GEHRIG D, LOQUERCIO A, DERPANIS K G, et al. End-to-end learning of representations for asynchronous event-based data[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019: 5632-5642.
[32] CANNICI M, CICCONE M, ROMANONI A, et al. A differentiable recurrent surface for asynchronous event-based data[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2020: 136-152.
[33] DENG Y, LI Y, CHEN H. AMAE: adaptive motion-agnostic encoder for event-based object classification[J]. IEEE Robotics and Automation Letters, 2020, 5(3): 4596-4603.
[34] DENG Y, CHEN H, LI Y. MVF-Net: a multi-view fusion network for event-based object classification[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2022, 32(12): 8275-8284.
[35] MESSIKOMMER N, GEHRIG D, LOQUERCIO A, et al. Event-based asynchronous sparse convolutional networks[C]//Proceedings of the European Conference on Computer Vision (ECCV), 2020: 415-431.
[36] MITROKHIN A, HUA Z Y, FERMULLER C, et al. Learning visual motion segmentation using event surfaces[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022: 14402-14411.
[37] TAHERKHANI A, BELATRECHE A, LI Y H, et al. A review of learning in biologically plausible spiking neural networks[J]. Neural Netw, 2020, 122: 253-272.
[38] BI Y, CHADHA A, ABBAS A, et al. Graph-based spatio-temporal feature learning for neuromorphic vision sensing[J]. IEEE Transactions on Image Processing, 2020, 29: 9084-9098.
[39] LI Z Q, NIKLAUS S, SNAVELY N, et al. Neural scene flow fields for space-time view synthesis of dynamic scenes[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 2021: 6494-6504.
[40] LI Y J, ZHOU H, YANG B B, et al. Graph-based asynchronous event processing for rapid object recognition[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 2021: 914-923.
[41] SCHAEFER S, GEHRIG D, SCARAMUZZA D. AEGNN: asynchronous event-based graph neural networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022: 12361-12371.
[42] SUN D, JI H. Event-based object detection using graph neural networks[C]//Proceedings of the IEEE 12th Data Driven Control and Learning Systems Conference (DDCLS), Xiangtan, China, 2023: 1895-1900.
[43] GALLEGO G, DELBRüCK T, ORCHARD G, et al. Event-based vision: a survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(1): 154-180.
[44] CULURCIELLO E, ETIENNE-CUMMINGS R, BOAHEN K A. A biomorphic digital image sensor[J]. IEEE J Solid-State Circuits, 2003, 38(2): 281-294.
[45] HUANG J, GUO M H, CHEN S S. A dynamic vision sensor with direct logarithmic output and full-frame picture-on-demand[C]//Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS), Baltimore, USA, 2017: 1-4.
[46] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems, 2012: 1097-1105.
[47] GALLEGO G, LUND J E A, MUEGGLER E, et al. Event-based, 6-DOF camera tracking from photometric depth maps[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(10): 2402-2412.
[48] KIM H, HANDA A, BENOSMAN R, et al. Simultaneous mosaicing and tracking with an event camera[C]//Proceedings of the British Machine Vision Conference (BMVC), UK, 2014: 1-12.
[49] SCHEERLINCK C, BARNES N, MAHONY R. Continuous-time intensity estimation using event cameras[C]//Proceedings of the Asian Conference on Computer Vision (ACCV), Perth, Australia, 2018: 308-324.
[50] ORCHARD G, MEYER C, ETIENNE-CUMMINGS R, et al. HFirst: a temporal approach to object recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(10): 2028-2040.
[51] ZHAO B, DING R, CHEN S, et al. Feedforward categorization on AER motion events using cortex-like features in a spiking neural network[J]. IEEE Transactions on Neural Networks and Learning Systems, 2015, 26(9): 1963-1978.
[52] KOSTA A K, APOLINARIO M P E, ROY K. Live demonstration: ANN vs SNN vs hybrid architectures for event-based real-time gesture recognition and optical flow estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Vancouver, BC, Canada, 2023: 4148-4149.
[53] BARDOW P, DAVISON A J, LEUTENEGGER S. Simultaneous optical flow and intensity estimation from an event camera[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016: 884-892.
[54] REBECQ H, RANFTL R, KOLTUN V, et al. Events-to-video: bringing modern computer vision to event cameras[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019: 3852-3861.
[55] GEHRIG M, SCARAMUZZA D. Recurrent vision Transformers for object detection with event cameras[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, 2023: 13884-13893.
[56] MASSA R, MARCHISIO A, MARTINA M, et al. An efficient spiking neural network for recognizing gestures with a DVS camera on the loihi neuromorphic processor[C]//Proceedings of the International Joint Conference on Neural Networks (IJCNN), 2020.
[57] CORDONE L, MIRAMOND B, THIERION P. Object detection with spiking neural networks on auto-motive event data[C]//Proceedings of the International Joint Conference on Neural Networks (IJCNN), Padua, Italy, 2022.
[58] CANNICI M, CICCONE M, ROMANONI A, et al. Event-based convolutional networks for object detection in neuromorphic cameras[J]. arXiv:1805.07931, 2018.
[59] WANG L, MOSTAFAVI I S M, HO Y S, et al. Event based high dynamic range image and very high frame rate video generation using conditional generative adversarial network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019: 10073-10082.
[60] LAGORCE X, ORCHARD G, GALLUPI F, et al. HOTS: a hierarchy of event-based time-surfaces for pattern recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(7): 1346-1359.
[61] AIMAR A, MOSTAFA H, CALABRESE E, et al. NullHop: a flexible convolutional neural network accelerator based on sparse representations of feature maps[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 30(3): 644-656.
[62] PEROT E, TOURNEMIRE P D, NITTI D, et al. Learning to detect objects with a 1 megapixel event camera[J]. arXiv:2009.13436, 2020.
[63] FANG W, YU Z, CHEN Y, et al. Deep residual learning in spiking neural networks [J]. arXiv:2102.04159, 2021.
[64] MAQUEDA A I, LOQUERCIO A, GALLEGO G, et al. Event-based vision meets deep learning on steering prediction for self-driving cars[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018: 5419-5427.
[65] ZHU A, YUAN L, CHANEY K, et al. Ev-flownet: self-supervised optical flow estimation for event-based cameras[J]. arXiv:1802.06898, 2018.
[66] DENG Y, CHEN H, LIU H, et al. A voxel graph CNN for object classification with event cameras[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022: 1162-1171.
[67] CANNICI M, CICCONE M, ROMANONI A, et al. Asynchronous convolutional networks for object detection in neuromorphic cameras[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Long Beach, CA, USA, 2019: 1656-1665.
[68] LIU M, DELBRUCK T. Adaptive time-slice block-matching optical flow algorithm for dynamic vision sensors[C]//Proceedings of the British Machine Vision Conference (BMVC), Newcastle, UK, 2018: 1-12.
[69] ROY K, JAISWAL A, PANDA P. Towards spike-based machine intelligence with neuromorphic computing[J]. Nature, 2019, 575(7784): 607-617.
[70] TOYOIZUMI T, PFISTER J, AIHARA K, et al. Spike-timing dependent plasticity and mutual information maximization for a spiking neuron model[C]//Advances in Neural Information Processing Systems, 2005.
[71] SAFA A, SAHLI H, BOURDOUX A, et al. Learning event-based spatio-temporal feature descriptors via local synaptic plasticity: a biologically-realistic perspective of computer vision[J]. arXiv:2111.00791, 2021.
[72] NEFTCI E, MOSTAFA H, ZENKE F. Surrogate gradient learning in spiking neural networks: bringing the power of gradient-based optimization to spiking neural networks[J]. IEEE Signal Processing Magazine, 2019, 36(6): 51-63.
[73] FANG W, YU Z, CHEN Y, et al. Incorporating learnable membrane time constant to enhance learning of spiking neural networks[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 2021.
[74] ORCHARD G, COHEN G, JAYAWANT A, et al. Converting static image datasets to spiking neuromorphic datasets using saccades[J]. Frontiers in Neuroscience, 2015, 9: 437.
[75] SERRANO-GOTARREDONA T, LINARES-BARRANCO B. Poker-DVS and MNIST-DVS. Their history, how they were made, and other details[J]. Frontiers in Neuromorphic Engineering. Frontiers in Neuroscience, 2015, 9: 481.
[76] KIM J, BAE J, PARK G, et al. N-ImageNet: towards robust, fine-grained object recognition with event cameras[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 2021: 2126-2136.
[77] LI H, LIU H, JI X, et al. Cifar10-dvs: an event-stream dataset for object classification[J]. Frontiers in Neuroscience, 2017, 11: 309.
[78] AMOS S, BRAMBILLA M, BOURDIS N, et al. HATS: histograms of averaged time surfaces for robust event-based object classification[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018: 1731-1740.
[79] BI Y, CHADHA A, ABBAS A, et al. Graph-based object classification for neuromorphic vision sensing[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019: 491-501.
[80] HWANG I, KIM J, KIM Y M. Ev-NeRF: event based neural radiance field[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), Waikoloa, HI, USA, 2023: 837-847.