Overview of Smoke and Fire Detection Algorithms Based on Deep Learning

doi:10.3778/j.issn.1002-8331.2206-0154

Abstract

Abstract: Among various disasters, fire is one of the main disasters that most often and universally threaten public safety and social development. With the rapid development of economic construction and the increasing size of cities, the number of major fire hazards has increased dramatically. However, the widely used smoke sensor method of fire detection is vulnerable to factors such as distance, resulting in untimely detection. The introduction of video surveillance systems has provided new ideas to solve this problem. Traditional image processing algorithms based on video are earlier proposed methods, and the recent rapid development of machine vision and image processing technologies has resulted in a series of methods using deep learning techniques to automatically detect fires in video and images, which have very important practical applications in the field of fire safety. In order to comprehensively analyze the improvements and applications related to deep learning methods for fire detection, this paper first briefly introduces the fire detection process based on deep learning, and then focuses on a detailed comparative analysis of deep methods for fire detection in three granularities：classification, detection, and segmentation, and elaborates the relevant improvements taken by each class of algorithms for existing problems. Finally, the problems of fire detection at the present stage are summarized and future research directions are proposed.

Key words: deep learning, fire detection, object classification, object detection, object segmentation

摘要： 在各种灾害中，火灾是最经常、最普遍的威胁公众安全和社会发展的主要灾害之一。随着经济建设的迅猛发展，城市规模日趋扩大，重大火灾隐患急剧增加。然而，目前广泛使用的烟雾传感器探测火灾的方法，易受距离等因素影响，导致检测不及时。视频监控系统的引入为解决这一问题提供了新思路，基于视频的传统图像处理算法是较早提出的方法，最近机器视觉与图像处理技术快速发展，涌现出一系列使用深度学习技术来自动检测视频和图像中火灾的方法，在消防安全领域具有非常重要的实际应用价值。为了综合分析火灾检测的深度学习方法相关改进及应用，简要介绍了基于深度学习的火灾检测流程，重点从分类、检测、分割3个粒度对火灾检测的深度方法详细对比分析，阐述每类算法针对现有问题采取的相关改进。总结现阶段火灾检测存在的问题，并提出未来的研究方向。

关键词: 深度学习, 火灾检测, 目标分类, 目标检测, 目标分割

ZHU Yuhua, SI Yiyi, LI Zhihui. Overview of Smoke and Fire Detection Algorithms Based on Deep Learning[J]. Computer Engineering and Applications, 2022, 58(23): 1-11.

祝玉华, 司艺艺, 李智慧. 基于深度学习的烟雾与火灾检测算法综述[J]. 计算机工程与应用, 2022, 58(23): 1-11.

References

[1] KUANG H L，CHAN L L H，YAN H.Multi-class fruit detection based on multiple color channels[C]//2015 International Conference on Wavelet Analysis and Pattern Recognition（ICWAPR），2015：1-7.
[2] LOWE D G.Distinctive image features from scale-invariant key points[J].International Journal of Computer Vision，2004，60（2）：91-110.
[3] LIENHART R，MAYDT J.An extended set of Haar-like features for rapid object detection[C]//International Conference on Image Processing，2002.
[4] KRIZHEVSKY A，SUTSKEVER I，HINTON G E.Image-Net classification with deep convolutional neural networks[C]//Advances in Neural Information Processing Systems，2012.
[5] SZEGEDY C，LIU W，JIA Y，et al.Going deeper with convolutions[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：1-9.
[6] SIMONYAN K，ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[J].arXiv：1409. 1556，2014.
[7] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[8] HU J，SHEN L，SUN G.Squeeze-and-excitation networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：7132-7141.
[9] REN S，HE K，GIRSHICK R，et al.Faster R-CNN：towards real-time object detection with region proposal networks[C]//Advances in Neural Information Processing Systems，2015.
[10] LIU W，ANGUELOV D，ERHAN D，et al.SSD：single shot multibox detector[C]//European Conference on Computer Vision.Cham：Springer，2016：21-37.
[11] ZHANG S，WEN L，BIAN X，et al.Single-shot refinement neural network for object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：4203-4212.
[12] REDMON J，FARHADI A.Yolov3：an incremental improvement[J].arXiv：1804.02767，2018.
[13] LONG J，SHELHAMER E，DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：3431-3440.
[14] RONNEBERGER O，FISCHER P，BROX T.U-net：convolutional networks for biomedical image segmentation[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention.Cham：Springer，2015：234-241.
[15] CHEN L C，PAPANDREOU G，KOKKINOS I，et al.Semantic image segmentation with deep convolutional nets and fully connected CRFs[J].arXiv：1412.7062，2014.
[16] FRIZZI S，KAABI R，BOUCHOUICHA M，et al.Convolutional neural network for video fire and smoke detection[C]//IECON 2016-42nd Annual Conference of the IEEE Industrial Electronics Society，2016：877-882.
[17] 冯路佳，王慧琴，王可，等.基于目标区域的卷积神经网络火灾烟雾识别[J].激光与光电子学进展，2020，57（16）：83-91.
FENG L J，WANG H Q，WANG K，et al.Convolutional neural network fire smoke detection based on target region[J].Laser & Optoelectronics Progress，2020，57（16）：83-91.
[18] YUAN F，ZHANG L，WAN B，et al.Convolutional neural networks based on multi-scale additive merging layers for visual smoke recognition[J].Machine Vision and Applications，2019，30（2）：345-358.
[19] JEON M，CHOI H S，LEE J，et al.Multi-scale prediction for fire detection using convolutional neural network[J].Fire Technology，2021：1-19.
[20] FILONENKO A，KURNIANGGORO L，JO K H.Smoke detection on video sequences using convolutional and recurrent neural networks[C]//International Conference on Computational Collective Intelligence.Cham：Springer，2017：558-566.
[21] NGUYEN M D，VU H N，PHAM D C，et al.Multistage real-time fire detection using convolutional neural networks and long short-term memory networks[J].IEEE Access，2021，9：146667-146679.
[22] JI S，XU W，YANG M，et al.3D convolutional neural networks for human action recognition[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2012，35（1）：221-231.
[23] 吴凡，王慧琴，王可.时空域深度学习火灾烟雾检测[J].液晶与显示，2021，36（8）：1186-1195.
WU F，WANG H Q，WANG K.Spatio-temporal deep learning fire smoke detection[J].Chinese Journal of Liquid Crystals and Displays，2021，36（8）：1186-1195.
[24] 陈俊周，汪子杰，陈洪瀚，等.基于级联卷积神经网络的视频动态烟雾检测[J].电子科技大学学报，2016，45（6）：992-996.
CHEN J Z，WANG Z J，CHEN H H，et al.Dynamic smoke detection using cascaded convolutional neural network for surveillance videos[J].Journal of University of Electronic Science and Technology of China，2016，45（6）：992-996.
[25] LUO Y，ZHAO L，LIU P，et al.Fire smoke detection algorithm based on motion characteristic and convolutional neural networks[J].Multimedia Tools and Applications，2018，77（12）：15075-15092.
[26] 殷亚萍，柴文，凌毅德，等.基于特征分析的卷积神经网络烟雾识别[J].无线电工程，2021，51（7）：526-533.
YIN Y P，CHAI W，LING Y D，et al.Convolutional neural network for smoke recognition based on feature analysis[J].Radio Engineering，2021，51（7）：526-533.
[27] 刘通，程江华，华宏虎，等.结合[YdUaVa]颜色模型和改进MobileNetV3的视频烟雾检测方法[J].国防科技大学学报，2021，43（5）：80-85.
LIU T，CHENG J H，HUA H H，et al.Video smoke detection method combining [YdUaVa] color model and improved MobileNetV3[J].Journal of National University of Defense Technology，2021，43（5）：80-85.
[28] RYU J，KWAK D.Flame detection using appearance-based pre-processing and convolutional neural network[J].Applied Sciences，2021，11（11）：5138.
[29] HU C，TANG P，JIN W D，et al.Real-time fire detection based on deep convolutional long-recurrent networks and optical flow method[C]//2018 37th Chinese Control Conference（CCC），2018：9061-9066.
[30] 蒋珍存，温晓静，董正心，等.基于深度学习的VGG16图像型火灾探测方法研究[J].消防科学与技术，2021，40（3）：375-377.
JIANG Z C，WEN X J，DONG Z X，et al.Research on VGG16 image fire detection method based on deep learning[J].Fire Science and Technology，2021，40（3）：375-377.
[31] YIN Z，WAN B，YUAN F，et al.A deep normalization and convolutional neural network for image smoke detection[J].IEEE Access，2017，5：18429-18438.
[32] XU Z，GUO Y，SALEH J H.Tackling small data challenges in visual fire detection：a deep convolutional generative adversarial network approach[J].IEEE Access，2020，9：3936-3946.
[33] 卫鑫，武淑红，王耀力.基于深度卷积长短期记忆网络的森林火灾烟雾检测模型[J].计算机应用，2019，39（10）：2883-2887.
WEI X，WU S H，WANG Y L.Forest fire smoke detection model based on deep convolution long short-term memory network[J].Journal of Computer Applications，2019，39（10）：2883-2887.
[34] QIANG X，ZHOU G，CHEN A，et al.Forest fire smoke detection under complex backgrounds using TRPCA and TSVB[J].International Journal of Wildland Fire，2021，30（5）：329-350.
[35] HE L，GONG X，ZHANG S，et al.Efficient attention based deep fusion CNN for smoke detection in fog environment[J].Neurocomputing，2021，434：224-238.
[36] 张坚鑫，郭四稳，张国兰，等.基于多尺度特征融合的火灾检测模型[J].郑州大学学报（工学版），2021，42（5）：13-18.
ZHANG J X，GUO S W，ZHANG G L，et al.Fire detection model based on multi-scale feature fusion[J].Journal of Zhengzhou University（Engineering Science），2021，42（5）：13-18.
[37] QIAN H，SHI F，CHEN W，et al.A fire monitoring and alarm system based on channel-wise pruned YOLOv3[J].Multimedia Tools and Applications，2021：1-19.
[38] XU R，LIN H，LU K，et al.A forest fire detection system based on ensemble learning[J].Forests，2021，12（2）：217.
[39] 栗俊杰，毛鹏军，淡文慧，等.基于YOLOv2-Tiny的无人机火灾检测与云台跟踪研究[J].消防科学与技术，2022，41（1）：108-112.
LI J J，MAO P J，DAN W H，et al.Research on UAV fire detection and PTZ tracking based on YOLOv2-Tiny[J].Fire Science and Technology，2022，41（1）：108-112.
[40] 晋耀，张为.采用Anchor-Free网络结构的实时火灾检测算法[J].浙江大学学报（工学版），2020，54（12）：2430-2436.
JIN Y，ZHANG W.Real-time fire detection algorithm with Anchor-Free network architecture[J].Journal of Zhejiang University（Engineering Science），2020，54（12）：2430-2436.
[41] ZHANG Q，LIN G，ZHANG Y，et al.Wild land forest fire smoke detection based on Faster R-CNN using synthetic smoke images[J].Procedia Engineering，2018，211：441-446.
[42] 孙婷婷.基于静态特征和动态行为的火灾检测模型[J].中国安全生产科学技术，2021，17（1）：96-101.
SUN T T.Fire detection model based on static characteristics and dynamic behavior[J].Journal of Safety Science and Technology，2021，17（1）：96-101.
[43] LIN G，ZHANG Y，XU G，et al.Smoke detection on video sequences using 3D convolutional neural networks[J].Fire Technology，2019，55（5）：1827-1847.
[44] 李澎林，章军伟，李伟.基于光流改进与YOLOv3的烟雾检测方法[J].浙江工业大学学报，2021，49（1）：9-15.
LI P L，ZHANG J W，LI W.Smoke detection method based on optical flow improvement and YOLOv3[J].Journal of Zhejiang University of Technology，2021，49（1）：9-15.
[45] PARK M J，KO B C.Two-step real-time night-time fire detection in an urban environment using static ELASTIC-YOLOv3 and temporal fire-tube[J].Sensors，2020，20（8）：2202.
[46] 谢书翰，张文柱，程鹏，等.嵌入通道注意力的YOLOv4火灾烟雾检测模型[J].液晶与显示，2021，36（10）：1445-1453.
XIE S H，ZHANG W Z，CHENG P，et al.Fire smoke detection model based on YOLOv4 with channel attention[J].Chinese Journal of Liquid Crystals and Displays，2021，36（10）：1445-1453.
[47] 刘丽娟，陈松楠.一种基于改进SSD的烟雾实时检测模型[J].信阳师范学院学报（自然科学版），2020，33（2）：305-311.
LIU L J，CHEN S N.Real-time smoke detection model based on improved SSD[J].Journal of Xinyang Normal University（Natural Science Edition），2020，33（2）：305-311.
[48] 高洁，王战红，刘纲.基于FSSD的微光烟雾检测方法[J].电子测量技术，2021，44（5）：123-128.
GAO J，WANG Z H，LIU G.Low light level smoke detection method based on FSSD[J].Electronic Measurement Technology，2021，44（5）：123-128.
[49] XU G，ZHANG Y，ZHANG Q，et al.Video smoke detection based on deep saliency network[J].Fire Safety Journal，2019，105：277-285.
[50] ZHOU Y C，HU Z Z，YAN K X，et al.Deep learning-based instance segmentation for indoor fire load recognition[J].IEEE Access，2021，9：148771-148782.
[51] KHAN S，MUHAMMAD K，HUSSAIN T，et al.Deepsmoke：deep learning model for smoke detection and segmentation in outdoor environments[J].Expert Systems with Applications，2021，182：115125.
[52] PAN J，OU X，XU L.A collaborative region detection and grading framework for forest fire smoke using weakly supervised fine segmentation and lightweight faster-RCNN[J].Forests，2021，12（6）：768.
[53] KO B C，PARK J O，NAM J Y.Spatiotemporal bag-of-features for early wildfire smoke detection[J].Image and Vision Computing，2013，31（10）：786-795.
[54] Cair.Fire-Detection-Image-Dataset[EB/OL].[2022-04-10].https：//github.com/cair/Fire-Detection-Image-Dataset.
[55] GEETHA S，ABHISHEK C S，AKSHAYANAT C S.Machine vision based fire detection techniques：a survey[J].Fire Technology，2021，57（2）：591-623.
[56] CHINO D Y T，AVALHAIS L P S，RODRIGUES J F，et al.BoWFire：detection of fire in still images by integrating pixel color and texture analysis[C]//2015 28th SIBGRAPI Conference on Graphics，Patterns and Images，2015：95-102.
[57] 余婕，田世祥，王伟，等.基于AHP-Bayes的城镇老旧小区动态智能化火灾风险评估模型——以上海市M小区为例[J].安全与环境工程，2021，28（5）：10-17.
YU J，TIAN S X，WANG W，et al.Dynamic intelligent fire risk assessment model for urban old community based on AHP-Bayes—taking old community M of Shanghai as an example[J].Safety and Environmental Engineering，2021，28（5）：10-17.
[58] 疏学明，颜峻，胡俊，等.基于Bayes网络的建筑火灾风险评估模型[J].清华大学学报（自然科学版），2020，60（4）：321-327.
SHU X M，YAN J，HU J，et al.Risk assessment model for building fires based on a Bayesian network[J].Journal of Tsinghua University（Science and Technology），2020，60（4）：321-327.
[59] 谢旭春.在建民用建筑火灾风险评估模型构建及应用[J].消防科学与技术，2020，39（8）：1100-1103.
XIE X C.Development and application of fire risk assessment model for civil buildings under construction[J].Fire Science and Technology，2020，39（8）：1100-1103.
[60] CRESWELL A，WHITE T，DUMOULIN V，et al.Generative adversarial networks：an overview[J].IEEE Signal Processing Magazine，2018，35（1）：53-65.
[61] HOWARD A，SANDLER M，CHU G，et al.Searching for MobileNetv3[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：1314-1324.
[62] LIU Z，LI J，SHEN Z，et al.Learning efficient convolutional networks through network slimming[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2736-2744.
[63] ZHAO Y，ZHANG H，ZHANG X，et al.Fire smoke detection based on target-awareness and depthwise convolutions[J].Multimedia Tools and Applications，2021：1-15.