Cross-Domain Adaptive Object Detection Based on CNN Image Enhancement in Foggy Conditions

doi:10.3778/j.issn.1002-8331.2211-0132

Abstract

Abstract: This paper proposes a cross-domain adaptive object detection method based on CNN（convolutional neural networks） image enhancement, which addresses the problem that the accuracy of target detection algorithm decreases due to the low quality of images captured by the visual perception system of autonomous vehicles in foggy conditions. An end-to-end target detection network is constructed, which integrates DIP（digital image processing） and CNN adaptive image enhancement module, to improve the image quality in foggy weather through a small CNN parameter predictor that learns enhancement parameters adaptively. Furthermore, a multi-scale DA（domain adaptive） module is connected to YOLOv4 backbone, which through adversarial training, reduces the domain differences caused by foggy conditions and increases the accuracy of target detection in foggy weather. In the stage of training, CNN, DA and YOLOv4 are learned in an end-to-end manner. In the stage of detection, both CNN and DA modules are removed, only using the images that pre-training weights have adaptively detected in normal and foggy weather, which will not increase the complexity of the original network and thus satisfy the timing requirement of autonomous vehicles. An experiment based on the open dataset Foggy Cityscapes indicates that the proposed method can significantly enhance the image quality in foggy weather, increasing the average accuracy of target detection by 10.4%, which effectively enhances the target detection ability of autonomous vehicles in foggy conditions.

Key words: autopilot, object detection, image enhancement, convolutional neural networks（CNN）, domain adaptation, YOLOv4

摘要： 针对自动驾驶车辆视觉感知系统在雾天条件下捕获图像质量较低，造成目标检测算法精度下降的问题，提出一种基于卷积神经网络（convolutional neural networks，CNN）图像增强的跨域自适应雾天目标检测方法。构建一个端到端目标检测网络，融合数字图像处理技术（digital image processing，DIP）和CNN的自适应图像增强模块，通过小型CNN参数预测器自适应学习增强参数，提升雾天图像质量；进一步地，将多尺度领域自适应（domain adaptive，DA）模块与YOLOv4主干网络相连，通过对抗训练减少由雾天造成的领域差异，提高雾天目标检测精度。在训练阶段，所提方法以端到端的方式学习CNN、DA模块以及YOLOv4，而在目标检测阶段将移除CNN及DA模块，仅使用预训练权重在正常天气和雾天天气自适应地检测图像，不会增加原有网络复杂性，从而保证自动驾驶车辆的实时性要求。在公开数据集Foggy Cityscapes上的实验表明，采用所提方法使雾天图像质量显著增强，目标检测平均精度提升了10.4%，有效提升了雾天条件下自动驾驶车辆对目标的识别能力。

关键词: 自动驾驶, 目标检测, 图像增强, 卷积神经网络（CNN）, 领域自适应, YOLOv4

GUO Ying, LIANG Ruilin, WANG Runmin. Cross-Domain Adaptive Object Detection Based on CNN Image Enhancement in Foggy Conditions[J]. Computer Engineering and Applications, 2023, 59(16): 187-195.

郭迎, 梁睿琳, 王润民. 基于CNN图像增强的雾天跨域自适应目标检测[J]. 计算机工程与应用, 2023, 59(16): 187-195.

References

[1] 朱向雷，王海弛，尤翰墨，等.自动驾驶智能系统测试研究综述[J].软件学报，2021，32（7）：2056-2077.
ZHU X L，WANG H C，YOU H M，et al.Survey on testing of intelligent systems in autonomous vehicles[J].Journal of Software，2021，32（7）：2056-2077.
[2] ZHOU J，GAO D，ZHANG D.Moving vehicle detection for automatic traffic monitoring[J].IEEE Transportation Vehicle Technology，2007，56（1）：51-59.
[3] MANDELOS N A，KERAMITSOGLOU I，KIRANOUDIS C T.A background subtraction algorithm for detecting and tracking vehicles[J].Expert System Apply，2011，38（3）：1619-1631.
[4] 茅智慧，朱佳利，吴鑫，等.基于YOLO的自动驾驶目标检测研究综述[J].计算机工程与应用，2022，58（15）：68-77.
MAO Z H，ZHU J L，WU X，et al.Review of YOLO based target detection for autonomous driving[J].Computer Engineering and Applications，2022，58（15）：68-77.
[5] 徐谦，李颖，王刚.基于深度学习的行人和车辆检测[J].吉林大学学报（工学版），2019，49（5）：1661-1667.
XU Q，LI Y，WANG G.Pedestrian-vehicle detection based on deep learning[J].Journal of Jilin University（Engineering Edition），2019，49（5）：1661-1667.
[6] 汪昱东，郭继昌，王天保.一种改进的雾天图像行人和车辆检测算法[J].西安电子科技大学学报，2020，47（4）：70-77.
WANG Y D，GUO J C，WANG T B.Algorithm for foggy-image pedestrian and vehicle detection[J].Journal of Xi’an University of Electronic Science and Technology，2020，47（4）：70-77.
[7] ZHU Q，MAI J，SHAO L.A fast single image haze removal algorithm using color attenuation prior[J].IEEE Transactions on Image Processing，2015，24（11）：3522-3533.
[8] LI B Y，PENG X L，WANG Z Y，et al.AOD-Net：all-in-one dehazing network[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision，Venice，2017：4780-4788.
[9] CAI B，XU X，JIA K，et al.DehazeNet：an end-to-end system for single image haze removal[J].IEEE Transactions on Image Processing，2016，25（11）：5187-5198.
[10] ENGIN D，GEN A，EKENEL H K.Cycle-Dehaze：enhanced cycleGAN for single image dehazing[C]//Proceedings of the 2018 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops，Salt Lake City，2018：938-946.
[11] BINKOWSKI M，SUTHERLAND D J，ARBEL M，et al.Demystifying MMD GANs[C]//Proceedings of the 6th International Conference on Learning Representations，Vancouver，2018：149806.
[12] HEUSEL M，RAMSAUER H，UNTERTHINER T，et al.GANs trained by a two time-scale update rule converge to a local Nash equilibrium[C]//Advances in Neural Information Processing Systems，2017：6626-6637.
[13] PAN S J，YANG Q J，ENGINEERING D.A survey on transfer learning[J].IEEE Transactions on Knowledge and Data Engineering，2009，22（10）：1345-1359.
[14] WEISS K，KHOSHGOFTAAR T M，WANG D J.A survey of transfer learning[J].Journal of Big Data，2016，3（1）：1-40.
[15] CHEN Y H，LI W，SAKARIDIS C，et al.Domain adaptive Faster R-CNN for object detection in the wild[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition，Salt Lake City，2018：3339-3348.
[16] GIRSHICK R.Fast R-CNN[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision，Santiago，2015：1440-1448.
[17] HE Z W，ZHANG L.Multi-adversarial Faster-RCNN for unrestricted object detection[C]//Proceedings of the 2019 IEEE International Conference on Computer Vision，Seoul，2019：6667-6676.
[18] SAITO K，USHIKU Y，HARADA T，et al.Strong-weak distribution alignment for adaptive object detection[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition，Long Beach，2019：6956-6965.
[19] CHEN Y，WANG H，LI W，et al.Scale-aware domain adaptive Faster R-CNN[J].International Journal of Computer Vision，2021，129（7）：2223-2243.
[20] REDMON J，DIVVALA S，GIRSHICK R，et al.You only look once：unified，real-time object detection[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition，2016：779-788.
[21] ZHANG S Z，TUO H Y，H J，et al.Domain adaptive YOLO for one-stage cross-domain detection[J].arXiv：2106.13939，2021.
[22] HNEWA M，RADHA H.Multiscale domain adaptive YOLO for cross-domain object detection[C]//Proceedings of the 2021 IEEE International Conference on Image Processing，Anchorage，2021：3323-3327.
[23] KHODAMBASHI S，MOGHADDAM M.An impulse noise fading technique based on local histogram processing[C]//Proceedings of the 2009 IEEE International Symposium on Signal Processing and Information Technology，Ajman，2009：95-100.
[24] PARIHAR A S，GUPTA Y K，SINGODIA Y，et al.A comparative study of image dehazing algorithms[C]//Proceedings of the 5th International Conference on Communication and Electronics Systems，Coimbatore，2020：766-771.
[25] LIU F，CAO L，SHAO X P，et al.Polarimetric dehazing utilizing spatial frequency segregation of images[J].Applied Optics，2015，54（27）：8116-8122.
[26] ZHANG W H，GAO S S，JING C，et al.Single image dehazing method under the influence of un-uniform illumination[C]//2016 IEEE 14th International Conference on Dependable，Autonomic & Secure Computing，International Conference on Pervasive Intelligence & Computing，International Conference on Big Data Intelligence & Computing & Cyber Science & Technology Congress，Auckland，2016：233-238.
[27] YU R S，LIU W Y，ZHANG Y S，et al.Deep exposure：learning to expose photos with asynchronously reinforced adversarial learning[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems，Montréal，2018：2153-2163.
[28] HU Y M，HE H，XU C X，et al.Exposure：a white-box photo post-processing framework[J].ACM Transactions on Graphics，2018，37（2）：26.
[29] ZENG H，CAI J R，LI L，et al.Learning image-adaptive 3D lookup tables for high performance photo enhancement in real-time[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2020，44（4）：2058-2073.
[30] HSU H K，YAO C H，TSAI Y H，et al.Progressive domain adaptation for object detection[C]//2020 IEEE Winter Conference on Applications of Computer Vision，Snowmass，2020：749-757.
[31] BOCHKOVSKIY A，WANG C Y，LIAO H Y M.YOLOv4：optimal speed and accuracy of object detection[J].arXiv：2004.10934，2020.
[32] SAKARIDIS C，DAI D，GOOL L V.Semantic foggy scene understanding with synthetic data[J].International Journal of Computer Vision，2018，126：973-992.
[33] HE K，SUN J，TANG X J，et al.Single image haze removal using dark channel prior[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2010，33（12）：2341-2353.
[34] 方帅，赵育坤，李心科，等.基于光照估计的夜间图像去雾[J].电子学报，2016，44（11）：7.
FANG S，ZHAO Y K，LI X K，et al.Nighttime haze removal based on illumination estimation[J].Acta Electronica Sinica，2016，44（11）：7.
[35] HUANG S C，LE T H，JAW D W.DSNet：joint semantic learning for object detection in inclement weather conditions[J].IEEE Transaction on Pattern Analysis and Machine Intelligence，2020，43（8）：2623-2633.
[36] HAUTIERE N，TAREL J P，AUBER D，et al.Blind contrast enhancement assessment by gradient ratioing at visible edges[J].Image Analysis & Stereology，2008，2：87-95.
[37] ZHANG L，ZHANG L，BOVIK A C.A feature-enriched completely blind image quality evaluator[J].IEEE Transactions on Image Processing，2015，24（8）：2579-2591.
[38] XIE R C，YU F，WANG J C，et al.Multi-level domain adaptive learning for cross-domain detection[C]//Proceedings of the 2019 IEEE International Conference on Computer Vision Workshops，Seoul，2019：3213-3219.
[39] ZHENG Y T，HUANG D，LIU S T，et al.Cross-domain object detection through coarse-to-fine feature adaptation[C]//Proceedings of the 2020 IEEE Computer Society Conference on Computer Vision and Pattern Recognition，Seattle，2020：13763-13772.