Research Progress of Ground Cloud Image Segmentation Method

doi:10.3778/j.issn.1002-8331.2210-0037

Abstract

Abstract: The change and distribution of cloud cover have an important influence on photovoltaic power forecast, astronomical telescope station location and weather forecast. Cloud-based observation is an important way of cloud observation, which mainly reflects the information of cloud base and the distribution, change and movement of clouds in local areas of the sky. Segmentation of foundation cloud image is the foundation of building an automatic observation system of foundation cloud image, so the related research is of great significance. Thanks to the rapid development of deep learning, the general semantic segmentation model has been continuously expanded and applied to the field of ground-based cloud image segmentation, and achieved good segmentation performance. However, considering that different types of clouds have different thicknesses and edges are difficult to distinguish, the segmentation method of ground cloud images based on deep learning still faces severe challenges in accuracy and efficiency. Firstly, from three aspects：threshold, traditional machine learning and deep learning, the segmentation methods of foundation cloud image are comprehensively summarized. Secondly, the data sets commonly used in the segmentation of foundation cloud images are summarized. In addition, the performances of various ground cloud image segmentation methods on GDNCI and WSISEG datasets are compared, and the advantages and disadvantages of various methods in the two datasets are analyzed. Finally, the problems to be solved and the future research direction in the segmentation of foundation cloud images are prospected.

Key words: segmentation of foundation cloud image, threshold segmentation, machine learning, deep learning, ground cloud image segmentation dataset

摘要： 云量的变化和分布对光伏发电功率预测、天文望远镜观测站选址和气象预报等均具有重要影响。地基云观测是云观测的重要方式，是对卫星云观测数据的有效补充，其主要反映天空局部区域云底信息和云层分布、变化及运动情况。地基云图分割是构建地基云图自动观测系统的基础，因此相关研究具有重要意义。得益于深度学习的飞速发展，深度卷积神经网络的通用语义分割模型被不断拓展应用到地基云图分割领域，并取得了良好分割性能。然而由于地基云图内在的特殊性和复杂性，特别是考虑到不同类别云层厚度不同并且边缘难以区分等问题，基于深度学习的地基云图分割方法仍面临着精度及效率等方面的严峻挑战。从阈值、传统机器学习和深度学习三方面出发，对地基云图分割方法进行全面综述；总结了当前地基云图分割常用的数据集；对比了各类地基云图分割方法在GDNCI和WSISEG两种数据集上的性能，并分析了各类方法在两种数据集中的优劣；最后进行了全面总结，并对地基云图分割中待解决的问题与未来的研究方向进行了展望。

关键词: 地基云图分割, 阈值分割, 机器学习, 深度学习, 地基云图分割数据集

SHI Chaojun, LI Xingkuan, ZHANG Ke, HAN Leile, YANG Shifang. Research Progress of Ground Cloud Image Segmentation Method[J]. Computer Engineering and Applications, 2023, 59(13): 1-16.

石超君, 李星宽, 张珂, 韩磊乐, 杨世芳. 地基云图分割方法研究进展[J]. 计算机工程与应用, 2023, 59(13): 1-16.

References

[1] ZHANG J L，LIU P，ZHANG F，SONG Q Q.CloudNet：ground‐based cloud classification with deep convolutional neural network[J].Geophysical Research Letters，2018，45（16）：8665-8672.
[2] STEPHENS G L.Cloud feedbacks in the climate system：a critical review[J].Journal of Climate，2005，18（2）：237-273.
[3] SCHNEIDER S H，WASHINGTON W M，CHERVIN R M.Cloudiness as a climatic feedback mechanism：effects on cloud amounts of prescribed global and regional surface temperature changes in the NCAR GCM[J].Journal of the Atmospheric Sciences，1978，35（12）：2207-2221.
[4] DUDA D P，MINNIS P，KHLOPENKOV K，et al.Estimation of 2006 Northern Hemisphere contrail coverage using MODIS data[J].Geophysical Research Letters，2013，40（3）：612-617.
[5] 卢静，翟海青，刘纯，等.光伏发电功率预测统计方法研究[J].华东电力，2010，38（4）：563-567.
LU J，ZHAI H Q，LIU C，et al.Study on statistical method for predicting photo voltaic generation power[J].East China Electric Power，2010，38（4）：563-567.
[6] 龚莺飞，鲁宗相，乔颖，等.光伏功率预测技术[J].电力系统自动化，2016，40（4）：140-151.
GONG Y F，LU Z X，QIAO Y，et al.An overview of photovoltaic energy system output forecasting technology[J].Automation of Electric Power Systems，2016，40（4）：140-151.
[7] VARELA A M，BERTOLIN C，MUNOZ B C，et al.Astronomical site selection：on the use of satellite data for aerosol content monitoring[J].Monthly Notices of the Royal Astronomical Society，2008，391（2）：507-520.
[8] ZHENG X，YE J，CHEN Y，et al.Detecting comma-shaped clouds for severe weather forecasting using shape and motion[J].IEEE Transactions on Geoscience and Remote Sensing，2019，57（6）：3788-3801.
[9] GLOTFELTY T，ALAPATY K，HE J，et al.The weather research and forecasting model with aerosol cloud interactions（WRF-ACI）：development，evaluation，and initial application[J].Monthly Weather Review，2019，147（5）：1491-1511.
[10] STEFANUT S，OLLERER K，MANOLE A，et al.National environmental quality assessment and monitoring of atmospheric heavy metal pollution—a moss bag approach[J].Journal of Environmental Management，2019，248：109224.
[11] MSCA C，PED B，AMB B，et al.Multiproxy analysis of a lateglacial-holocene sedimentary section in the fuegian steppe（northern tierra del fuego，argentina）：implications for coastal landscape evolution in relation to climatic variability and sea-level fluctuations-ScienceDirect[J].Palaeogeography，Palaeoclimatology，Palaeoecology，2020，557：109941.
[12] FLUKE C J，JACOBS C.Surveying the reach and maturity of machine learning and artificial intelligence in astronomy[J].Wiley Interdisciplinary Reviews：Data Mining and Knowledge Discovery，2020，10（2）：e1349.
[13] KIM N，NA S I，PARK C W，et al.An artificial intelligence approach to prediction of corn yields under extreme weather conditions using satellite and meteorological data[J].Applied Sciences，2020，10（11）：3785.
[14] CHEN Y，YANG Y，LIU C，et al.A hybrid application algorithm based on the support vector machine and artificial intelligence：an example of electric load forecasting[J].Applied Mathematical Modelling，2015，39（9）：2617-2632.
[15] SWANN J A，SILVERMANN G M，LINDEMANN E A，et al.Artificial intelligence facilitates performance review and characterization of prehospital emergency medical services treatment[J].Journal of the American College of Surgeons，2020，231（4）：305-306.
[16] VNA C，HG B，IKU B.Artificial intelligence based ensemble model for prediction of vehicular traffic noise[J].Environmental Research，2020，180：108852.
[17] MDF A，FP B.Assessing bank efficiency and performance with operational research and artificial intelligence techniques：a survey[J].European Journal of Operational Research，2010，204（2）：189-198.
[18] MATEO-GARCíA G，LAPARRA V，DAN L P，et al.Transferring deep learning models for cloud detection between Landsat-8 and Proba-V[J].ISPRS Journal of Photogrammetry and Remote Sensing，2020，160（4）：1-17.
[19] LU J，WANG Y，ZHU Y，et al.P_SegNet and NP_SegNet：new neural network architectures for cloud recognition of remote sensing images[J].IEEE Access，2019，7（1）：87323-87333.
[20] 陆雅君，陈刚毅，龚克坚，等.测云方法研究进展[J].气象科技，2012，40（5）：689-697.
LU Y J，CHEN G Y，GONG K J，et al.Research progress of cloud measurement method[J].Meteorological Science and Technology，2012，40（5）：689-697.
[21] SCHIFFER R A，ROSSOW W B.ISCCP global radiance data set：a new resource tor climate research[J].Bulletin of the American Meteorological Society，1985，66（12）：1498-1505.
[22] 高太长，刘磊，赵世军，等.全天空测云技术现状及进展[J].应用气象学报，2010，21（1）：101-109.
GAO T C，LIU L，ZHAO S J，et al.The actuality and progress of whole sky cloud sounding techniques[J].Journal of Applied Meteorological Science，2010，21（1）：101-109.
[23] LONG C N，SABBURG J M，CALBO J，et al.Retrieving cloud characteristics from ground-based daytime color all-sky images[J].Journal of Atmospheric and Oceanic Technology，2006，23（5）：633-652.
[24] LIU L Y，LV B L，XU J Y，et al.Automatic observation experiments of cloud amounts and cloud forms based on the image recognition[C]//Proceedings of 2019 International Conference on Meteorology Observations（ICMO），2019：1-4.
[25] AN N W，KAICUN.A comparison of MODIS-derived cloud fraction with surface observations at five SURFRAD sites[J].Journal of Applied Meteorology and Climatology，2015，54（5）：1009-1020.
[26] CALBO J，SABBURG J.Feature extraction from whole-sky ground-based images for cloud-type recognition[J].Journal of Atmospheric and Oceanic Technology，2008，25（1）：3-14.
[27] PFISTER G，MCKENZIE R L，LILEY J B，et al.Cloud coverage based on all-sky imaging and its impact on surface solar irradiance[J].Journal of Applied Meteorology，2003，42（10）：1421-1434
[28] 吕达仁，霍娟，吕曜.地基全天空成像仪遥感的科学，技术问题和初步试验[M]//中国遥感——奋进创新20年.北京：气象出版社，2001：114-120.
LYU D R，HUO J，LYU Y.Science，technical problems and preliminary experiment of ground-based all-sky imager remote sensing[M]//China remote sensing-20 years of innovation.Beijing：Meteorological Press，2001：114-120.
[29] 霍娟，吕达仁，王越.全天空云识别阈值法的数值模拟初步研究[J].自然科学进展，2006，16（4）：480-484.
HUO J，LYU D R，WANG Y.Preliminary study on numerical simulation of threshold method for all-sky cloud recognition[J].Advances in Natural Science，2006，16（4）：480-484.
[30] HEINLE A，MACKE A，SRIVASTAV A.Automatic cloud classification of whole sky images[J].Atmospheric Measurement Techniques，2010，3（3）：557-567.
[31] NETO S L M，WANGENHEIM R V，PERIERA E B，et al. The use of euclidean geometric distance on rgb color space for the classification of sky and cloud patterns[J].Journal of Atmospheric and Oceanic Technology，2010，27（9）：1504-1517.
[32] OTSU N.A thresholding selection method from gray-level histogram[J].IEEE SMC，1978，8：62-66.
[33] 杨俊，吕伟涛，马颖，等.基于自适应阈值的地基云自动检测方法[J].应用气象学报，2009，20（6）：713-721.
YANG J，LU W T，MA Y，et al.An automatic ground-based cloud detection method based on adaptive threshold[J].Journal of Applied Meteorological Science，2009，20（6）：713-721.
[34] 杨俊，吕伟涛，马颖，等.基于局部阈值插值的地基云自动检测方法[J].气象学报，2010（6）：1007-1017.
YANG J，LU W T，MA Y，et al.An automatic ground-based cloud detection method based on the local threshold interpolation[J].Acta Meteorologica Sinica，2010（6）：1007-1017.
[35] LI Q，LU W，YANG J.A hybrid thresholding algorithm for cloud detection on ground-based color images[J].Journal of Atmospheric and Oceanic Technology，2011，28（10），1286-1296.
[36] SHI C，WANG Y，WANG C，et al.Ground-based cloud detection using graph model built upon superpixels[J].IEEE Geoscience & Remote Sensing Letters，2017，14（5）：719-723.
[37] ACHANTA R，SHAJI A，SMITH K，et al.SLIC superpixels compared to state-of-the-art superpixel methods[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2012，34（11）：2274-2282.
[38] CSURKA G，DANCE C，FAN L，et al.Visual categorization with bags of keypoints[C]//Workshop on Statistical Learning in Computer Vision，2004：1-2.
[39] RIGATTI S J.Random forest[J].Journal of Insurance Medicine，2017，47（1）：31-39.
[40] DEV S，LEE Y H，WINKLER S.Color-based segmentation of sky/cloud images from ground-based cameras[J].IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing，2016，10（1）：231-242.
[41] MAS J F，FLORES J J.The application of artificial neural networks to the analysis of remotely sensed data[J].International Journal of Remote Sensing，2008，29（3）：617-663.
[42] MOUNTRAKIS G，IM J，OGOLE C.Support vector machines in remote sensing：a review[J].ISPRS Journal of Photogrammetry and Remote Sensing，2011，66（3）：247-259.
[43] TARAVAT A，DEL F F，CORNARO C，et al.Neural networks and support vector machine algorithms for automatic cloud classification of whole-sky ground-based images[J].IEEE Geoscience and Remote Sensing Letters，2014，12（3）：666-670.
[44] BLAZEK M，PATA P.Colour transformations and K-means segmentation for automatic cloud detection[J].Meteorologische Zeitschrift，2015，24（5）：503-509.
[45] KRAUZ L，JANOUT P，BLA?EK M，et al.Assessing cloud segmentation in the chromacity diagram of all-sky images[J].Remote Sensing，2020，12（11）：1902.
[46] RUDRAPPA G，VIJAPUR A.Cloud classification using K-means clustering and content based image retrieval technique[C]//Proceedings of 2020 International Conference on Communication and Signal Processing（ICCSP），2020.
[47] YE L，CAO Z G，XIAO Y，et al.Supervised fine-grained cloud detection and recognition in whole-sky images[J].IEEE Transactions on Geoscience and Remote Sensing，2019，57（10）：7972-7985.
[48] CHEN E，WU X，WANG C.Application of improved convolutional neural network in image classification[C]//Proceedings of 2019 International Conference on Machine Learning，Big Data and Business Intelligence（MLBDBI），2019：109-113.
[49] LONG J，SHELHSMER E，DARRELL T.Fully convolutional networks for semantic segmentation[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2015：3431-3440.
[50] 毋立芳，贺娇瑜，简萌，等.局部聚类分析的FCN-CNN云图分割方法[J].软件学报，2018，29（4）：1049-1059.
WU L F，HE J Y，JIAN M，et al.Local clustering analysis based FCN-CNN for cloud image segmentation[J].Journal of Software，2018，29（4）：1049-1059.
[51] SHI C J，ZHOU Y T，QIU B，et al.Diurnal and nocturnal cloud segmentation of all-sky imager（ASI） images using enhancement fully convolutional networks[J].Atmospheric Measurement Techniques，2019，12（9）：4713-4724.
[52] RONNEBERGER O，FISCHER P，VROX T，et al.U-Net：convolutional networks for biomedical image segmentation[C]//Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention，2015：234-241.
[53] DEV S，MANANDHR S，LEE Y H.Multi-label cloud segmentation using a deep network[C]//Proceedings of 2019 USNC-URSI Radio Science Meeting（Joint with AP-S Symposium），2019：113-114.
[54] SHI C J，ZHOU Y T，QIU B，et al.CloudU-Net：a deep convolutional neural network architecture for daytime and nighttime cloud images’ segmentation[J].IEEE Geoscience and Remote Sensing Letters，2020，18（10）：1688-1692.
[55] CHEN L C，PAPANDREOU G，KOKKINOS I，et al.DeepLab：Semantic image segmentation with deep convolutional nets，atrous convolution，and fully connected crfs[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，40（4）：834-848.
[56] ZHANG M，LUCAS J，BA J，et al.Lookahead optimizer：k steps forward，1 step back[C]//Advances in Neural Information Processing Systems，2019：1-12.
[57] SHI C J，ZHOU Y T，QIU B.CloudU-Netv2：a cloud segmentation method for ground-based cloud images based on deep learning[J].Neural Processing Letters，2021，53（4）：2715-2728.
[58] FU J，LIU J，TIAN H，et al.Dual attention network for scene segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition，2019：3146-3154.
[59] SHI C J，ZHOU Y T，QIU B.CloudRaedNet：residual attention-based encoder-decoder network for ground-based cloud images segmentation in nychthemeron[J].International Journal of Remote Sensing，2022，43（6）：2059-2075.
[60] HE K，ZHANG X，REN S，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[61] BADRINARAYANAN V，KENDALL A，CIPOLLA R.SegNet：a deep convolutional encoder-decoder architecture for image segmentation[J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2017，39（12）：2481-2495.
[62] DEV S，NAUTIYAL A，LEE Y H，et al.CloudSegNet：a deep network for nychthemeron cloud image segmentation[J].IEEE Geoscience and Remote Sensing Letters，2019，16（12）：1814-1818.
[63] LIN M，CHEN Q，YAN S.Network in network[J].arXiv：1312.4400，2013.
[64] XIE W，LIU D，YANG M，et al.SegCloud：a novel cloud image segmentation model using a deep convolutional neural network for ground-based all-sky-view camera observation[J].Atmospheric Measurement Techniques，2020，13（4）：1953-1961.
[65] LOFFE S，SZEGEDY C.Batch normalization：accelerating deep network training by reducing internal covariate shift[C]//Proceedings of International Conference on Machine Learning，2015：448-456.
[66] 张雪，贾克斌，刘钧，等.面向轻量化的地基云图分割技术研究[J].测控技术，2022（9）：264-270.
ZHANG X，JIA K B，LIU J，et al.Segmentation technology of ground-based cloud image for lightweight[J].Measurement and Control Technology，2022（9）：264-270.
[67] ZHANG Z，YANG S，LIU S，et al.Ground-based cloud detection using multiscale attention convolutional neural network[J].IEEE Geoscience and Remote Sensing Letters，2021，19：1-5.
[68] YU F，KOLTUN V.Multi-scale context aggregation by dilated convolutions[J].arXiv：1511.07122，2015.
[69] ZHOU Z，ZHANG F，XIAO H.A novel ground-based cloud image segmentation method by using deep transfer learning[J].IEEE Geoscience and Remote Sensing Letters，2021，19：1-5.
[70] DIANNE G，WILIEM A，LOVELL B C.Deep-learning from mistakes：automating cloud class refinement for sky image segmentation[C]//Proceedings of 2019 Digital Image Computing：Techniques and Applications（DICTA），2019：1-8.
[71] LIU S，ZHANG J，ZHANG Z.TransCloudSeg：ground-based cloud image segmentation with transformer[J].IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing，2022，15：6121-6132.
[72] FABEL Y，NOURI B，WILBERT S，et al.Applying self-supervised learning for semantic cloud segmentation of all-sky images[J].Atmospheric Measurement Techniques Discussions，2022，15（3）：797-809.
[73] DEV S，SAVOY F M，LEE Y H，et al.Nighttime sky/cloud image segmentation[C]//Proceedings of 2017 IEEE International Conference on Image Processing（ICIP），2017：345-349.
[74] DEV S，SAVOY F M，LEE Y H，et al.WAHRSIS：a low-cost high-resolution whole sky imager with near-infrared capabilities[J].arXiv：1605.06595，2016.
[75] ZHANG Z，WANG S，LIU S，et al.Ground-based remote sensing cloud detection using dual pyramid network and encoder decoder constraint[J].IEEE Transactions on Geoscience and Remote Sensing，2022，60：1-10.
[76] CHEN L C，ZHU Y，PAPANDREOU G，et al.Encoder-decoder with atrous separable convolution for semantic image segmentation[C]//Proceedings of the European Conference on Computer Vision（ECCV），2018：801-818.