Survey of Object Detection Methods Under Adverse Weather Conditions

doi:10.3778/j.issn.1002-8331.2112-0405

Abstract

Abstract: Driven by deep learning, object detection methods have made great progress in recent years. However, under adverse weather conditions, such as foggy, dust, rain, snow and other weather conditions, the images will be degraded and blurred, which is difficult to extract effective features and cause problems to the later object detection. This paper summarizes the research of domestic and foreign scholars on object detection tasks in adverse weather in recent years, and analyzes it from three aspects：the establishment and use of relevant datasets, the research of image restoration algorithms for adverse weather conditions and their impact on object detection tasks, and the application of domain adaptive methods of transfer learning for object detection. Meanwhile, the paper also compares the experimental results of relevant methods, and puts forward the research focus in the future.

Key words: deep learning, object detection, adverse weather, image restoration, domain adaptation

摘要： 在深度学习的推动下，目标检测方法在近些年取得了很大的进展。但在特殊天气条件下，如在常见的雾霾、沙尘、雨天、雪天等天气条件下拍摄到的图像会退化模糊，难以提取有效特征，从而对后期的目标检测带来巨大的困难。对近年来国内外学者在特殊天气下目标检测的研究进行了总结归纳，从相关数据集的建立和用途、面向特殊天气条件下的图像恢复算法研究及其对目标检测任务的影响、迁移学习中的领域自适应方法在目标检测的应用三个方面进行分析。对相关方法的实验结果进行综合比较，并提出今后的研究重点。

关键词: 深度学习, 目标检测, 特殊天气, 图像恢复, 迁移学习

LIN Meng, ZHOU Gang, YANG Yawei, SHI Jun. Survey of Object Detection Methods Under Adverse Weather Conditions[J]. Computer Engineering and Applications, 2022, 58(13): 36-47.

林猛, 周刚, 杨亚伟, 石军. 特殊天气条件下的目标检测方法综述[J]. 计算机工程与应用, 2022, 58(13): 36-47.

References

[1] DALAL N，TRIGGS B.Histograms of oriented gradients for human detection[C]//Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2005：886-893.
[2] VIOLA P，JONES M.Rapid object detection using a boosted cascade of simple features[C]//Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition，2001.
[3] TIAN S，PAN Y，HUANG C，et al.Text flow：A unified text detection system in natural scene images[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：4651-4659.
[4] LIU S，QI L，QIN H，et al.Path aggregation network for instance segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：8759-8768.
[5] NOH H，HONG S，HAN B.Learning deconvolution network for semantic segmentation[C]//Proceedings of the IEEE International Conference on Computer Vision，2015：1520-1528.
[6] KANG K，LI H，YAN J，et al.T-CNN：Tubelets with convolutional neural networks for object detection from videos[J].IEEE Transactions on Circuits and Systems for Video Technology，2017，28（10）：2896-2907.
[7] KRIZHEVSKY A，SUTSKEVER I，GEOFFREY E H.Image net classification with deep convolutional neural networks[J].Advances in Neural Information Processing Systems，2012，25：1097-1105.
[8] GIRSHICK R，DONAHUE J，DARRELL T，et al.Rich feature hierarchies for accurate object detection and semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2014：580-587.
[9] LIN T Y，DOLLáR P，GIRSHICK R，et al.Feature pyramid networks for object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：2117-2125.
[10] REDMON J，FARHADI A.You only look once：Unified，real-time object detection[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：779-788.
[11] LIU W，ANGUELOV D，ERHAN D，et al.SSD：Single shot multibox detector[C]//Proceedings of the European Conference on Computer Vision，2016：21-37.
[12] LIN T Y，GOYAL P，GIRSHICK R，et al.Focal loss for dense object detection[C]//Proceedings of the IEEE International Conference on Computer Vision，2017：2980-2988.
[13] ZOU Z，SHI Z，GUO Y，et al.Object detection in 20 years：A survey[J].arXiv：1905.05055，2019.
[14] AGARWAL S，TERRAIL J O D，JURIE F J A P A.Recent advances in object detection in the age of deep convolutional neural networks[J].arXiv：1809.03193，2018.
[15] ZHAO Z Q，ZHENG P，XU S T，et al.Object detection with deep learning：A review[J].IEEE Transaction on Neural Networks and Learning Systems，2019，30（11）：3212-3232.
[16] LIU L，OUYANG W，WANG X，et al.Deep learning for generic object detection：A survey[J].International Journal of Computer Vision，2020，128（2）：261-318.
[17] 陈飞，刘云鹏，李思远.复杂环境下的交通标志检测与识别方法综述[J].计算机工程与应用，2021，57（16）：65-73.
CHEN F，LIU Y P，LI S Y.Survey of traffic sign detection and recognition methods in complex environment[J].Computer Engineering and Applications，2021，57（16）：65-73.
[18] 董天天，曹海啸，阚希，等.复杂天气下交通场景多目标识别方法研究[J].信息通信，2020（11）：72-74.
DONG T T，CAO H X，KAN X，et al.Multi target recognition method of traffic scene in complex weather[J].Information & Communications，2020（11）：72-74.
[19] LIU W，HOU X，DUAN J，et al.End-to-end single image fog removal using enhanced cycle consistent adversarial networks[J].IEEE Transactions on Image Processing，2020，29：7819-7833.
[20] ZHAO S，ZHANG L，HUANG S，SHEN Y，et al.Dehazing evaluation：Real?world benchmark datasets，criteria，and baselines[J].IEEE Transactions on Image Processing，2020，29：6947-6962.
[21] TAREL J P，HAUTIERE N，CORD A，et al.Improved visibility of road scene images under heterogeneous fog[C]//Proceedings of the 2010 IEEE Intelligent Vehicles Symposium，2010：478-485.
[22] TAREL J P，HAUTIERE N，CARAFFA L，et al.Vision enhancement in homogeneous and heterogeneous fog[J].IEEE Intelligent Transportation Systems Magazine，2012，4（2）：6-20.
[23] ANCUTI C，ANCUTI C O，DE VLEESCHOUWER C.D-HAZY：A dataset to evaluate quantitatively dehazing algorithms[C]//Proceedings of the 2016 IEEE International Conference on Image Processing（ICIP），2016：2226-2230.
[24] ANCUTI C，ANCUTI C O，TIMOFTE R，et al.I-HAZE：A dehazing benchmark with real hazy and haze-free indoor images[C]//Proceedings of the International Conference on Advanced Concepts for Intelligent Vision Systems，2018：620-631.
[25] ANCUTI C O，ANCUTI C，TIMOFTE R，et al.O-HAZE：A dehazing benchmark with real hazy and haze-free outdoor images[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops，2018：754-762.
[26] LI B，REN W，FU D，et al.Benchmarking single-image dehazing and beyond[J].IEEE Transactions on Image Processing，2018，28（1）：492-505.
[27] HAHNER M，DAI D，SAKARIDIS C，et al.Semantic understanding of foggy scenes with purely synthetic data[C]//Proceedings of the 2019 IEEE Intelligent Transportation Systems Conference（ITSC），2019：3675-3681.
[28] SAKARIDIS C，DAI D，GOOL L.Semantic foggy scene understanding with synthetic data[J].International Journal of Computer Vision，2018，126（9）：973-992.
[29] LIU Y F，JAW D W，HUANG S C，et al.DesnowNet：Context-aware deep network for snow removal[J].IEEE Transactions on Image Processing，2018，27（6）：3064-3073.
[30] HU X，FU C W，ZHU L，et al.Depth-attentional features for single-image rain removal[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：8022-8031.
[31] WANG T，YANG X，XU K，et al.Spatial attentive single-image deraining with a high quality real rain dataset[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：12270-12279.
[32]?ISRA?L?H，?KASTEN?F.Koschmieders?theorie?der?horizontalen　sichtweite[M]//Die　Sichtweite　im　Nebel und die　M?glichkeien ihrer künstlichen Beeinflussung.Wiesbaden：VS Verlag?für?Sozialwissenschaften，1959：7-10.
[33] CORDTS M，OMRAN M，RAMOS S，et al.The cityscapes dataset for semantic urban scene understanding[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：3213-3223.
[34] WRENNINGE M，UNGER J.Synscapes：A photorealistic synthetic dataset for street scene parsing[J].arXiv：1810. 08705，2018.
[35] GARG K，NAYAR S K.Vision and rain[J].International Journal of Computer Vision，2007，75（1）：3-27.
[36] REN W，LIU S，ZHANG H，et al.Single image dehazing via multi-scale convolutional neural networks[C]//Proceedings of the European Conference on Computer Vision，2016：154-169.
[37] SILBERMAN N，HOIEM D，KOHLI P，et al.Indoor segmentation and support inference from RGBD images[C]//Proceedings of the European Conference on Computer Vision，2012：746-760.
[38] TREMBLAY M，HALDER S，DE CHARETTE R，et al.Rain rendering for evaluating and improving robustness to bad weather[J].International Journal of Computer Vision，2021，129（2）：341-360.
[39] CAESAR H，BANKITI V，LANG A H，et al.NuScenes：A multimodal dataset for autonomous driving[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：11621-11631.
[40] HALDER S，LALONDE J F，CHARETTE R D.Physics-based rendering for improving robustness to rain[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：10203-10212.
[41] DE CHARETTE R，TAMBURO R，BARNUM P C，et al.Fast reactive control for illumination through rain and snow[C]//Proceedings of the 2012 IEEE International Conference on Computational Photography（ICCP），2012：1-10.
[42] ISOLA P，ZHU J Y，ZHOU T，et al.Image-to-image translation with conditional adversarial networks[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：1125-1134.
[43] HUANG S W，LIN C T，CHEN S P，et al.AugGAN：Cross domain adaptation with GAN-based data augmentation[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：718-731.
[44] ZHAI L，JUEFEI-XU F，GUO Q，et al.It’s raining cats or dogs? adversarial rain attack on DNN perception[J].arXiv：2009.09205，2020.
[45] SHAO Y，LI L，REN W，et al.Domain adaptation for image dehazing[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：2808-2817.
[46] YASARLA R，SINDAGI V A，PATEL V M.Syn2Real transfer learning for image deraining using Gaussian processes[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：2726-2736.
[47] LI Y，YOU S，BROWN M S，et al.Haze visibility enhancement：A survey and quantitative benchmarking[J].Computer Vision and Image Understanding，2017，165：1-16.
[48] XU Y，WEN J，FEI L，et al.Review of video and image defogging algorithms and related studies on image restoration and enhancement[J].IEEE Access，2015，4：165-188.
[49] SINGH D，KUMAR V J.A comprehensive review of computational dehazing techniques[J].Archives of Computational Methods in Engineering，2019，26（5）：1395-1413.
[50] GUI J，CONG X，CAO Y，et al.A comprehensive survey on image dehazing based on deep learning[J].arXiv：2106.03323，2021.
[51] WANG H，WU Y，LI M，et al.A survey on rain removal from video and single image[J].arXiv：1909.08326，2019.
[52] HE K，SUN J，TANG X J，et al.Single image haze removal using dark channel prior[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2010，33（12）：2341-2353.
[53] HE K，SUN J，TANG X J，et al.Guided image filtering[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2012，35（6）：1397-1409.
[54] PENG Y T，LU Z，CHENG F C，et al.Image haze removal using air light white correction，local light filter，and aerial perspective prior[J].IEEE Transactions on Circuits and Systems for Video Technology，2019，30（5）：1385-1395.
[55] YANG D，SUN J.Proximal dehaze-net：A prior learning-based deep network for single image dehazing[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：702-717.
[56] YANG W，TAN R T，WANG S，et al.Single image deraining：From model-based to data-driven and beyond[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2020，43（11）：4059-4077.
[57] LI B，PENG X，WANG Z，et al.An all-in-one network for dehazing and beyond[J].arXiv：1707.06543，2017.
[58] HUANG S C，LE T H，JAW D W.DSNet：Joint semantic learning for object detection in inclement weather conditions[J].IEEE Transaction on Pattern Analysis and Machine Intelligence，2020，43（8）：2623-2633.
[59] LI C，GUO C，GUO J，et al.PDR-Net：Perception-inspired single image dehazing network with refinement[J].IEEE Transactions on Multimedia，2019，22（3）：704-716.
[60] REN S，HE K，GIRSHICK R，et al.Faster R-CNN：Towards real-time object detection with region proposal networks[J].Advance in Neural Information Processing Systems，2015，28：91-99.
[61] ZHANG H，SINDAGI V，PATEL V M，et al.Image de-raining using a conditional generative adversarial network[J].IEEE Transactions on Circuits and Systems for Video Technology，2019，30（11）：3943-3956.
[62] FU X，LIANG B，HUANG Y，et al.Lightweight pyramid networks for image deraining[J].IEEE Transactions on Neural Networks and Learning Systems，2019，31（6）：1794-1807.
[63] FAN Z，WU H，FU X，et al.Residual-guide feature fusion network for single image deraining[J].arXiv：1804.07493，2018.
[64] JIANG K，WANG Z，YI P，et al.Multi-scale progressive fusion network for single image deraining[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：8346-8355.
[65] LI S，ARAUJO I B，REN W，et al.Single image deraining：A comprehensive benchmark analysis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：3838-3847.
[66] PAN S J，YANG Q J，ENGINEERING D.A survey on transfer learning[J].IEEE Transactions on Knowledge and Data Engineering，2009，22（10）：1345-1359.
[67] WEISS K，KHOSHGOFTAAR T M，WANG D J.A survey of transfer learning[J].Journal of Big data，2016，3（1）：1-40.
[68] GOPALAN R，LI R，CHELLAPPA R.Domain adaptation for object recognition：An unsupervised approach[C]//Proceedings of the 2011 International Conference on Computer Vision，2011：999-1006.
[69] MANSOUR Y，MOHRI M.Domain adaptation：Learning bounds and algorithms[J].arXiv：0902.3430，2009.
[70] BEN-DAVID S，BLITZER J，CRAMMER K，et al.A theory of learning from different domains[J].Machine Learning，2010，79（1）：151-175.
[71] CHEN Y，LI W，SAKARIDIS C，et al.Domain adaptive Faster R-CNN for object detection in the wild[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：3339-3348.
[72] CHEN Y，WANG H，LI W，et al.Scale-aware domain adaptive Faster R-CNN[J].International Journal of Computer Vision，2021，129（7）：2223-2243.
[73] HE Z，ZHANG L.Multi-adversarial Faster R-CNN for unrestricted object detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：6668-6677.
[74] ZHU X，PANG J，YANG C，et al.Adapting object detectors via selective cross-domain alignment[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：687-696.
[75] SAITO K，USHIKU Y，HARADA T，et al.Strong-weak distribution alignment for adaptive object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：6956-6965.
[76] ZHAO G，LI G，XU R，et al.Collaborative training between region proposal localization and classification for domain adaptive object detection[C]//Proceedings of the European Conference on Computer Vision，2020：86-102.
[77] XIE R，WANG J.Multi-level domain adaptive learning for cross-domain detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops，2019.
[78] PAN Y S，MA A J，GAO Y，et al.Multi-scale adversarial cross-domain detection with robust discriminative lear-ning[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision，2020：1324-1332.
[79] SHEN Z，MAHESHWARI H，YAO W，et al.SCL：Towards accurate domain adaptive object detection via gradient detach based stacked complementary losses[J].arXiv：1911.02559，2019.
[80] CAI Q，PAN Y，NGO C W，et al.Exploring object relation in mean teacher for cross-domain detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：11457-11466.
[81] TARVAINEN A，VALPOLA H J.Mean teachers are better role models：Weight-averaged consistency targets improve semi-supervised deep learning results[J].arXiv：1703.01780，2017.
[82] DENG J，LI W，CHEN Y，et al.Unbiased mean teacher for cross-domain object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：4091-4101.
[83] HNEWA M，RADHA H.Multiscale domain adaptive yolo for cross-domain object detection[C]//Proceedings of the 2021 IEEE International Conference on Image Processing（ICIP），2021：3323-3327.
[84] WANG Y，ZHANG R，ZHANG S，et al.Domain-specific suppression for adaptive object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2021：9603-9612.
[85] KHODABANDEH M，VAHDAT A，RANJBAR M，et al.A robust learning approach to domain adaptive object detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：480-490.
[86] SINDAGI V A，OZA P，ASARLA R，et al.Prior-based domain adaptive object detection for hazy and rainy conditions[C]//Proceedings of the European Conference on Computer Vision，2020：763-780.
[87] LIU W，REN G，YU R，et al.Image-adaptive YOLO for object detection in adverse weather conditions[J].arXiv：2112.08088，2021.
[88] KIM T，JEONG M，KIM S，et al.Diversify and match：A domain adaptive representation learning paradigm for object detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：12456-12465.
[89] SHAN Y，LU W F，CHEW C.Pixel and feature level based domain adaptation for object detection in autonomous driving[J].Neurocomputing，2019，367：31-38.
[90] HSU H K，YAO C H，TSAI Y H.et al.Progressive domain adaptation for object detection[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision，2020：749-757.
[91] CHEN C，ZHENG Z，DING X，et al.Harmonizing transferability and discriminability for adapting object detectors[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：8869-8878.
[92] SOVIANY P，IONESCU R T，ROTA P，et al.Curriculum self-paced learning for cross-domain object detection[J].Computer Vision and Image Understanding，2021，204：103166.
[93] KIM Y，CHO D，HAN K，et al.Domain adaptation without source data[J].arXiv：2007.01524，2020.
[94] LI R，JIAO Q，CAO W，et al.Model adaptation：Unsupervised domain adaptation without source data[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2020：9641-9650.
[95] LI X，CHEN W，XIE D，et al.A free lunch for unsupervised domain adaptive object detection without source data[J].arXiv：2012.05400，2020.
[96] ZHANG H，PATEL V M.Densely connected pyramid dehazing network[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2018：3194-3203.
[97] LIU X，MA Y，SHI Z，et al.GridDehazeNet：Attention-based multi-scale network for image dehazing[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision，2019：7314-7323.
[98] WANG J，LAN C，LIU C，et al.Generalizing to unseen domains：A survey on domain generalization[J].arXiv：2103.03097，2021.