Application and Prospect of Python Language in Field of Hydrology and Water Resources

doi:10.3778/j.issn.1002-8331.2212-0008

Abstract

Abstract: Python programming language has gradually become a promising data analysis tool that can be widely used in various fields. However, in the field of hydrology and water resources, there is little research on using Python language for scientific analysis. Firstly, in this thesis, the common Python libraries used in the field of hydrology and water resources are introduced, based on the main research direction and application scenarios of Python language, in this thesis, the main research contents of it in the field of hydrology and water resources are summarized from four aspects：web crawler, data analysis, in-depth learning and Web development. Then, the common algorithms of deep learning used in the field are also summarized. Finally, from the prospects of automatic prediction, edge computation, virtual and augmented reality, reinforcement learning and transfer learning, it is expected to promote the rapid development of the field of hydrology and water resources with the advanced computer technology realized by Python language.

Key words: Python language, hydrology and water resources, deep learning, Web development, Web crawler, data analysis

摘要： Python编程语言逐渐成为各领域中应用前景广阔的数据分析工具。然而，在水文水资源领域中利用Python语言进行科学分析的研究较少。介绍了常用于水文水资源领域的Python库；基于Python语言的主要研究方向和应用场景，从网络爬虫、数据分析、深度学习和Web开发4个方面综述了Python语言在水文水资源领域的主要研究内容；归纳了深度学习运用在水文水资源领域的常见算法；从自动预测、边缘计算、虚拟现实技术、强化学习和迁移学习等方面进行了展望，期望以Python语言实现的前沿计算机技术为动力，促进水文水资源领域的快速发展。

关键词: Python语言, 水文水资源, 深度学习, Web开发, 网络爬虫, 数据分析

JIANG Qiuxiang, GUO Weipeng, WANG Zilong, OUYANG Xingtao, LONG Ruirui. Application and Prospect of Python Language in Field of Hydrology and Water Resources[J]. Computer Engineering and Applications, 2023, 59(9): 46-58.

姜秋香, 郭伟鹏, 王子龙, 欧阳兴涛, 隆睿睿. Python语言在水文水资源领域中的应用与展望[J]. 计算机工程与应用, 2023, 59(9): 46-58.

References

[1] 明日科技.Python从入门到精通[M].北京：清华大学出版社，2018.
Tomorrow’s Technology.Python goes from beginner to proficient[M].Beijing：Tsinghua University Press，2018.
[2] 左其亭，张志卓，马军霞.水资源分类方案及学科体系[J].资源科学，2021，43（11）：2215-2223.
ZUO Q T，ZHANG Z Z，MA J X.Classification scheme of water resources and its disciplinary system[J].Resources Science，2021，43（11）：2215-2223.
[3] 水利部印发关于推进智慧水利建设的指导意见和实施方案[J].水利建设与管理，2022，42（1）：5.
Ministry of Water Resources issued guidance opinions and implementation plans on promoting intelligent water conservancy construction[J].Water Conservancy Construction and Management，2012，42（1）：5.
[4] 道格·赫尔曼.Python3 标准库[M].北京：机械工业出版社，2018.
DOUG H.Python3 standard library[M].Beijing：China Machine Press，2018.
[5] HARRIS C R，MILLMAN K J，VAN DER WALT S J，et al.Array programming with NumPy[J].Nature，2020，585（7825）：357-362.
[6] 李庆辉.深入浅出Pandas[M].北京：机械工业出版社，2021.
LI Q H.Understanding Pandas in a simple way[M].Beijing：China Machine Press，2021.
[7] HAO J G，HO T K.Machine learning made easy：a review of scikit learn package in Python programming language[J].Journal of Educational and Behavioral Statistics，2019，44（3）：348-361.
[8] HUNTER J D.Matplotlib：a 2D graphics environment[J].Computing in Science & Engineering，2020，9（3）：90-95.
[9] GRATTAROLA D，ALIPPI C.Graph neural networks in TensorFlow and keras with spektral[J].IEEE Computational Intelligence Magazine，2021，16（1）：99-106.
[10] 胡松涛.Python网络爬虫实战[M].北京：清华大学出版社，2017.
HU S T.Python Web crawler practice[M].Beijing：Tsinghua University Press，2017.
[11] MOHAN S，SOLANKI A K，TALUJA H K，et al.Predicting the impact of the third wave of COVID-19 in India using hybrid statistical machine learning models：a time series forecasting and sentiment analysis approach[J].Computers in Biology and Medicine，2022，144：105354.
[12] VIRTANEN P，GOMMERS R，OLIPHANT T E，et al.Author correction：SciPy 1.0：fundamental algorithms for scientific computing in Python[J].Nature Methods，2020，17（3）：352-352.
[13] KRONHEIM B S，KUCHERA M P，PROSPER H B.TensorBNN：Bayesian inference for neural networks using TensorFlow[J].Computer Physics Communications，2022，270：108168.
[14] CAMP C H.PyMCR：a python library for multivariatecurve resolution analysis with alternating regression（MCR-AR）[J].Journal of Research of the National Institute of Standards and Technology，2019，124：18.
[15] 代红震.基于STAF和Python混合编程的自动化测试系统的研究和实现[D].南京：南京邮电大学，2015.
DAI H Z.Research and implementation of automated test system based on STAF and Python mixed programming[D].Nanjing：Nanjing University of Posts and Telecommunications，2015.
[16] 施林松.基于Selenium可视化的自动测试框架设计与实现[D].北京：中国科学院大学（中国科学院工程管理与信息技术学院），2017.
SHI L S.Design and realization of automatic test framework based on selenium[D].Beijing：University of Chinese Academy of Sciences（School of Engineering Management and Information Technology，Chinese Academy of Sciences），2017.
[17] GOODE K，HOFMANN H.Visual diagnostics of an explainer model：tools for the assessment of LIME explanations[J].Statistical Analysis and Data Mining：The ASA Data Science Journal，2021，14（2）：185-200.
[18] 王官云，任小强.基于Python语言的LTE业务差小区数据处理方法研究[J].工业控制计算机，2022，35（4）：115-116.
WANG G Y，REN X Q.Research on data processing method of LTE service cell based on Python language[J].Industrial Control Computer，2022，35（4）：115-116.
[19] 全力，杜景林，杨迎新.WRF模式气象数据可视化平台设计与实现[J].现代电子技术，2021，44（18）：87-91.
QUAN L，DU J L，YANG Y X.Design and implementation of meteorological data visualization platform in WRF model[J].Modern Electronics Technique，2021，44（18）：87-91.
[20] JANEZ D，TOMAZ C，ALES E，et al.Orange：data mining toolbox in Python[J].Journal of Machine Learning Research，2013，14（1）：2349-2353.
[21] PERSSON I，KHOJASTEH J.Python packages for exploratory factor analysis[J].Structural Equation Modeling：A Multidisciplinary Journal，2021，28（6）：983-988.
[22] 杜超.基于Python的聚焦爬虫的初步设计与实现[J].现代制造技术与装备，2020，56（12）：30-31.
DU C.Preliminary design and implementation of focused crawler based on Python[J].Modern Manufacturing Technology and Equipment，2020，56（12）：30-31.
[23] 余豪士，匡芳君.基于Python的反反爬虫技术分析与应用[J].智能计算机与应用，2018，8（4）：112-115.
YU H S，KUANG F J.Crawler and anti-anti-crawler technology based on Python[J].Intelligent Computer and Application，2018，8（4）：112-115.
[24] 蔡能健，刘东浩，邓辉，等.基于Web.py的MUSER软件系统界面实现[J].天文研究与技术，2017，14（2）：229-235.
CAI N J，LIU D H，DENG H，et al.The application method of Web.py in developing muser software system[J].Astronomical Research and Technology，2017，14（2）：229-235.
[25] 唐松，陈智铨.Python网络爬虫从入门到实践[M].北京：机械工业出版社，2017.
TANG S，CHEN Z Q.Python Web crawler from introduction to practice[M].Beijing：China Machine Press，2017.
[26] 吕鸿，吴泽宁，管新建，等.缺资料城市洪灾损失函数构建方法及应用[J].水科学进展，2021，32（5）：707-716.
LYU H，WU Z N，GUAN X J，et al.Construction method and applications of flood loss functions for cities lacking data[J].Advance in Water Science，2021，32（5）：707-716.
[27] WU Z，SHEN Y，WANG H，et al.Quantitative assessment of urban flood disaster vulnerability based on text data：case study in Zhengzhou[J].Water Supply，2020，20（2）：408-415.
[28] MA B，WU Z，WANG H，et al.Study on the classification of urban waterlogging rainstorms and rainfall thresholds in cities lacking actual data[J].Water，2020，12（12）.
[29] ZHANG J，ZOU T，LAI Y.Novel method for industrial sewage outfall detection：water pollution monitoring based on web crawler and remote sensing interpretation techniques[J].Journal of Cleaner Production，2021，312：127640.
[30] GUAN G L，WANG Y G，YANG L，et al.water-quality assessment and pollution-risk early-warning system based on web crawler technology and LSTM[J].International Journal of Environmental Research and Public Health，2022，19（18）.
[31] 陶皖.云计算与大数据[M].西安：西安电子科技大学出版社，2017：44.
TAO W.Cloud computing and big data[M].Xi’an：Xidian University Press，2017：44.
[32] MCKINNEY W.Python for data analysis[M].Beijing：China Machine Press，2018.
[33] 邱勇，毕怀斌，田宇心，等.污水处理厂进水数据特征识别与案例分析[J].环境科学学报，2021，42（4）：44-52.
QIU Y，BI H B，TIAN Y X，et al.Characterization model of influent data for wastewater treatment plants and its application in case studies[J].Acta Science Circumstantiae，2021，42（4）：44-52.
[34] ZHANG J，SHENG Y，CHEN W，et al.Design and analysis of a water quality monitoring data service platform[J].Computers，Materials & Continua，2021，66（1）：389-405.
[35] QIU Y G，XIE H，DUAN H T，et al.A novel spatiotemporal data model for river water quality visualization and analysis[J].IEEE Access，2019，7：155455-155461.
[36] DENG W，WANG G.A novel water quality data analysis framework based on time-series data mining[J].Journal of Environmental Management，2017，196：365-375.
[37] MANJAKKAL L，MITRA S，PETILLOT Y R，et al.Connected sensors，innovative sensor deployment，and intelligent data analysis for online water quality monitoring[J].IEEE Internet of Things Journal，2021，8（18）：13805-13824.
[38] KARHADE A V，SCHWAB J H.Introduction to the spine journal special issue on artificial intelligence and machine learning[J].The Spine Journal，2021，21（10）：1601-1603.
[39] 冯平，张立斌，李建柱，等.目标函数对水文模型参数敏感性和径流模拟不确定性影响分析[J/OL].水资源保护：1-15[2022-11-08].https：//kns.cnki.net/kcms/detail/32.1356.TV.20220519.1658.010.html.
FENG P，ZHANG L B，LI J Z，et al.The influence of objective function on the sensitivity of hydrological model parameters and the uncertainty of runoff simulation[J/OL].Water Resources Protection：1-15[2022-11-08].https：//kns.cnki.net/kcms/detail/32.1356.TV.20220519.1658.
010.html.
[40] 周志华.机器学习及其应用[M].北京：清华大学出版社，2007.
ZHOU Z H.Machine learning and its applications[M].Beijing：Tsinghua University Press，2007.
[41] HINTON G E，OSINDERO S，TEH Y W.A fast learning algorithm for deep belief nets[J].Neural Computation，2006，18（7）：1527-1554.
[42] MD Z A，TAREK M T，CHRIS Y，et al.A state-of-the-art survey on deep learning theory and architectures[J].Electronics，2019，8（3）：292.
[43] GOODFELLOW I BENGIO Y COURVILLE A.Deep learning[M].Cambridge，MA：MIT Press，2016.
[44] PEREIRA FILHO A J，DOS SANTOS C C.Modeling a densely urbanized watershed with an artificial neural network，weather radar and telemetric data[J].Journal of Hydrology，2006，317（1/2）：31-48.
[45] KOURGIALAS N N，DOKOU Z，KARATZAS G P.Statistical analysis and ANN modeling for predicting hydrological extremes under climate change scenarios：the example of a small Mediterranean agro-watershed[J].Journal of Environmental Management，2015，154：86-101.
[46] SULAIMAN M，EL-SHAFIE A，KARIM O，et al.Improved water level forecasting performance by using optimal steepness coefficients in an artificial neural network[J].Water Resources Management，2011，25（10）：2525-2541.
[47] SANKARANARAYANAN S，PRABHAKAR M，SATISH S，et al.Flood prediction based on weather parameters using deep learning[J].Journal of Water and Climate Change，2020，11（4）：1766-1783.
[48] CAI Y，ZHENG W，ZHANG X，et al.Research on soil moisture prediction model based on deep learning[J].Plos One，2019，14（4）：e0214508.
[49] KARPATNE A，ATLURI G，FAGHMOUS J H，et al.Theory-guided data science：a new paradigm for scientific discovery from data[J].IEEE Transactions on Knowledge and Data Engineering，2017，29（10）：2318-2331.
[50] WANG N，ZHANG D，CHANG H，et al.Deep learning of subsurface flow via theory-guided neural network[J].Journal of Hydrology，2020，584：124700.
[51] LE CUN Y.Deep learning[J].Nature，2015，521（7553）：436-444.
[52] AMIT S N K B，SHIRAISHI S，INOSHITA T，et al.Analysis of satellite images for disaster detection[C]//IEEE International Geoscience and Remote Sensing Symposium，2016：5189-5192.
[53] SONG C M.Application of convolution neural networks and hydrological images for the estimation of pollutant loads in ungauged watersheds[J].Water，2021，13（2）：239.
[54] 王立，吴羽溪，王小艺，等.基于遥感图像3D-CNN及氮磷循环的水华预测[J].控制理论与应用，2021，38（10）：1683-1691.
WANG L，WU Y X，WANG X Y，et al.Prediction of water bloom based on remote sensing image 3D-CNN and nitrogen and phosphours cycles[J].Control Theory and Applications，2021，38（10）：1683-1691.
[55] VOEGTLIN T，DOMINEY P F.Linear recursive distributed representations[J].Neural Networks，2005，18（7）：878-895.
[56] LI L，JIANG P，XU H，et al.Water quality prediction based on recurrent neural network and improved evidence theory：a case study of Qiantang River，China[J].Environmental Science and Pollution Research，2019，26（19）：19879-19896.
[57] WEI C，CHENG J.Nearshore two-step typhoon wind-wave prediction using deep recurrent neural networks[J].Journal of Hydroinformatics，2020，22（2）：346-367.
[58] CHIANG Y M，HAO R N，LI N N，et al.Water level forecasting by static and dynamic neural networks[J].Journal of Tianjin University，2017，50（3）：245-254.
[59] HOCHREITER S，SCHMIDHUBER J.Long short-term memory[J].Neural Computation，1997，9（8）：1735-1780.
[60] OLAH C.Understanding LSTM networks[EB/OL].（2015-08）.https：//colah.github.io/posts/2015-08-Understanding-LSTMs.
[61] KRATZERT F，KLOTZ D，BRENNER C，et al.Rainfall-runoff modelling using long short-term memory（LSTM） networks[J].Hydrology and Earth System Sciences，2018，22（11）：6005-6022.
[62] OUMA Y O，CHERUYOT R，WACHERA A N，et al.Rainfall and runoff time-series trend analysis using LSTM recurrent neural network and wavelet neural network with satellite-based meteorological data：case study of Nzoia hydrologic basin[J].Complex & Intelligent Systems，2021，8（1）：213-236.
[63] FANG K，SHEN C，KIFER D，et al.Prolongation of SMAP to spatiotemporally seamless coverage of continental U.S. using a deep learning neural network[J].Geophysical Research Letters，2017，44（21）：11030-11039.
[64] JEONG J，PARK E，CHEN H，et al.Estimation of groundwater level based on the robust training of recurrent neural networks using corrupted data[J].Journal of Hydrology，2020，582：124512.
[65] 刘亚新，樊启祥，尚毅梓，等.基于LSTM神经网络的水电站短期水位预测方法[J].水利水电科技进展，2019，39（2）：56-60.
LIU Y X，FAN Q X，SHANG Y Z，et al.Short-term water level prediction method of hydropower station based on LSTM neural network[J].Advances in Science and Technology of Water Resources，2019，39（2）：56-60.
[66] MU L，ZHENG F F，TAO R L，et al.Hourly and daily urban water demand predictions using a long short-term memory based model[J].Journal of Water Resources Planning and Management，2020，146（9）：05020017.
[67] KUEHNERT C，GONUGUNTLA N M，KRIEG H，et al.Application of LSTM networks for water demand prediction in optimal pump control[J].Water，2021，13（5）：644.
[68] ZANFEI A，BRENTAN B M.A short-term water demand forecasting model using multivariate long short-term memory with meteorological data[J].Journal of Hydroinformatics，2022，24（5）：1053-1065.
[69] LIU P，WANG J，SANGAIAH A，et al.Analysis and prediction of water quality using LSTM deep neural networks in IoT environment[J].Sustainability，2019，11（7）：2058.
[70] BAEK S，PYO J，CHUN J A.Prediction of water level and water quality using a CNN-LSTM combined deep learning approach[J].Water，2020，12（12）：1-13.
[71] ZOU Q，XIONG Q，LI Q，et al.A water quality prediction method based on the multi-time scale bidirectional long short-term memory network[J].Environmental Science and Pollution Research International，2020，27（14）：16853-16864.
[72] YU Z，YANG K，LUO Y，et al.Spatial-temporal process simulation and prediction of chlorophyll-a concentration in Dianchi Lake based on wavelet analysis and long-short term memory network[J].Journal of Hydrology，2020，582：124488.
[73] CHO K，MERRIENBOER B，BAHDANAU D.On the properties of neural machine translation：encoder-decoder approaches[J].arXiv：1409.1259，2014.
[74] ZHANG D，LINDHOLM G，RATNAWEERA H.Use long short-term memory to enhance internet of things for combined sewer overflow monitoring[J].Journal of Hydrology，2018，556：409-418.
[75] YU J X，ZHANG X，XU L L，et al.A hybrid CNN-GRU model for predicting soil moisture in maize root zone[J].Agricultural Water Management，2021，245：106649.
[76] KILINC H C，YURTSEVER A.Short-term streamflow forecasting using hybrid deep learning model based on grey wolf algorithm for hydrological time series[J].Sustainability，2022，14（6）.
[77] HINTON G E.Training products of experts by minimizing contrastive divergence[J].Neural Computation，2002，14（8）：1771-1800.
[78] UPADHYA V，SASTRY P S.An overview of restricted boltzmann machines[J].Journal of the Indian Institute of Science，2019，99（2）：225-236.
[79] LE ROUX N，BENJIO Y.Representational power of restricted Boltzmann machines and deep belief networks[J].Neural Computation，2008，20（6）：1631-1649.
[80] NIU G O，YI X H，CHEN C，et al.A novel effluent quality predicting model based on genetic-deep belief network algorithm for cleaner production in a full-scale paper-making wastewater treatment[J].Journal of Cleaner Production，2020，265：121787.
[81] SUNG J H，RYU Y，CHUNG E.Estimation of water-use rates based on hydro-meteorological variables using deep belief network[J].Water，2020，12（10）.
[82] CHEN Y，LIU G D，HUANG X H，et al.Groundwater remediation design underpinned by coupling evolution algorithm with deep belief network surrogate[J].Water Resources Management，2022，36（7）：2223-2239.
[83] 雷伟军，高寅生，冯永亮，等.网页设计与制作[M].北京：人民邮电出版社，2017：262.
LEI W J，GAO Y S，FENG Y L，et al.Web design and production[M].Beijing：Posts and Telecommunications Press，2017：262.
[84] TOMLINSON J E，ARNOTT J H，HAROU J J.A water resource simulator in Python[J].Environmental Modelling & Software，2020，126：104635.
[85] GRASSO S，CLAPS P，GANORA D，et al.A web‐based open‐source geoinformation tool for regional water resources assessment[J].Water Resources Management，2021，35（2）：675-687.
[86] EVANS S W，JONES N L，WILLIAMS G P，et al.Groundwater level mapping tool：an open source web application for assessing groundwater sustainability[J].Environmental Modelling & Software，2020，131：104782.
[87] BUSTAMANTE G R，NELSON E J，AMES D P，et al.Water data explorer：an open-source web application and Python library for water resources data discovery[J].Water，2021，13（13）：1850.
[88] PABLO B D A，KLEBER I S D S，PEDRO L B C.A web-based tool for synthesis assessments of the impacts of climate change on water resources[J].Environmental Modelling and Software，2020，133.
[89] CHEN J S，RAN X K.Deep learning with edge computing：a review[J].Proceedings of the IEEE，2019，107（8）：1655-1674.
[90] 刘洋.基于边缘计算的数据获取与处理系统设计与实现[D].北京：北京工业大学，2018.
LIU Y.Design and implementation of data acquisition and processing system based on edge computing[D].Beijing：Beijing University of Technology，2018.
[91] ZHOU J，CHEN Y，XIAO F，et al.Water quality prediction method based on transfer learning and echo state network[J].Journal of Circuits Systems and Computers，2021，30（14）：2150262.