Power dispatch data integration model based on Spark

doi:10.3778/j.issn.1002-8331.1609-0098

Computer Engineering and Applications ›› 2017, Vol. 53 ›› Issue (19): 65-70.DOI: 10.3778/j.issn.1002-8331.1609-0098

Previous Articles Next Articles

Power dispatch data integration model based on Spark

QU Zhaoyang1, CHEN Hexin1, HU Kewei2, LIU Yaowei3, DU Jianhong4

1.College of Information Engineering, Northeast Dianli University, Jilin, Jilin 132012, China
2.Control Center, State Grid Jilin Electric Power Co. , Ltd. , Changchun 130000, China
3.Jilin Power Supply Company, State Grid Jilin Electric Power Co. , Ltd. , Jilin, Jinlin 132000, China
4.Fengman Power Plant in Jilin City, Jilin, Jinlin 132113, China

Online:2017-10-01 Published:2017-10-13

基于Spark的电力调度数据整合模型

曲朝阳1，陈贺新1，胡可为2，刘耀伟3，独健鸿4

1.东北电力大学信息工程学院，吉林吉林 132012
2.国网吉林省电力有限公司调控中心，长春 130000
3.国网吉林省电力有限公司吉林供电公司，吉林吉林 132000
4.吉林市丰满发电厂，吉林吉林 132113

Abstract

Abstract: With the concept of data applying in the electric power industry, the construction of electric power dispatching data warehouse is foundation to support the unified data platform in the power dispatching center, in view of the multi-source data integration in the power dispatching center data warehouse there are redundant and inconsistent problems. This paper puts forward a kind of power dispatch data integration model based on Spark. First of all, the design of parallel forward maximum matching to remove redundancy algorithm, it filters the multiple system redundant data; then it gives data consistency processing method of correlation degree-oriented. It judges link of data according to the angle value of the feature vector, and repairs the inconsistent data. Based on the data integration of electric power dispatching center, it verifies the practicability of the data integration model.

Key words: power dispatch center, correlation degree, feature vector, data integration, Spark platform

摘要： 随着大数据理念在电力行业的应用，构建电力调度数据仓库是支撑电力调度中心统一数据平台的基础，针对电力调度中心的数据仓库将多源数据整合时面临的重复冗余和不一致问题，提出一种基于Spark的电力调度数据整合模型。设计并行化正向最大匹配去冗算法，对多个系统内冗余数据进行过滤操作；给出面向关联度的数据一致性处理方法，依据特征向量的夹角余弦值判断数据间的联系，进而对不一致数据修复。通过对某电力调度中心的数据进行整合实验，验证了该数据整合模型的可行性。

关键词: 电力调度中心, 关联度, 特征向量, 数据整合, Spark平台

QU Zhaoyang1, CHEN Hexin1, HU Kewei2, LIU Yaowei3, DU Jianhong4. Power dispatch data integration model based on Spark[J]. Computer Engineering and Applications, 2017, 53(19): 65-70.

曲朝阳1，陈贺新1，胡可为2，刘耀伟3，独健鸿4. 基于Spark的电力调度数据整合模型[J]. 计算机工程与应用, 2017, 53(19): 65-70.

[1]	LI Junli. Parallel Mutual-Information Computation of Categorical Data Based on Spark [J]. Computer Engineering and Applications, 2021, 57(7): 95-100.
[2]	ZHONG Yating, LIN Yanmei, CHEN Dingjia, PENG Yuzhong, ZENG Yuanpeng. Review on Integration Analysis and Application of Multi-omics Data [J]. Computer Engineering and Applications, 2021, 57(23): 1-17.
[3]	YANG Shu, SU Fang. Distributed Data Security Integrated Application System Based on Microservices [J]. Computer Engineering and Applications, 2021, 57(18): 238-247.
[4]	CHE Minshi, NIE Chunyan, YANG Chengjin, RUAN Xinlei, FAN Rujun. Research on Correlation Between Physiological Signal and Emotion Based on Grey Relational Method [J]. Computer Engineering and Applications, 2021, 57(11): 168-172.
[5]	BAO Shun, XU Xin, XIAO Jian, ZHOU Ligang. Pythagorean Hesitant Fuzzy Grey Correlation Multi-attribute Decision Making Method Based on Prospect Theory [J]. Computer Engineering and Applications, 2020, 56(18): 119-123.
[6]	LIU Jiayao, WANG Jiabin. Improvement of Slope One Algorithm and Its Implementation on Big Data Platform [J]. Computer Engineering and Applications, 2020, 56(1): 83-91.
[7]	ZHU Hongfeng, XIONG Wei, CUI Yaqi, LV Yafei. Research on IMM Algorithm of New Adaptive Turn Model [J]. Computer Engineering and Applications, 2019, 55(17): 252-258.
[8]	WANG Bin1, WANG Zhechen2, ZHOU Wei1, HAO Tianpeng1. Intuitionistic fuzzy decision-making method based on entropy and improved co-correlation degree [J]. Computer Engineering and Applications, 2018, 54(6): 247-251.
[9]	LI Gefei1, MA Weiyin2, LI Li3. Convex hull problem based on Spark platform [J]. Computer Engineering and Applications, 2018, 54(22): 67-73.
[10]	ZHU Dayong1，2, GUO Xing1，2, WU Jianguo1，2. Action recognition method using kinect 3D skeleton data [J]. Computer Engineering and Applications, 2018, 54(20): 152-158.
[11]	XIANG Jiang1, CHEN Yibing2, YANG Yi3. Research and design of public digitized cultural sharing cloud service platform [J]. Computer Engineering and Applications, 2018, 54(13): 258-265.
[12]	LIU Xuebo, LI Deng’ao, ZHAO Jumin, LIAO Shujian. J wave auto-detection based on variable step size support vector machine [J]. Computer Engineering and Applications, 2017, 53(23): 203-207.
[13]	LIN Hui1, YU Ruipeng2, WANG Lijuan1, SHAO Congying1. Study on object-oriented automatic classification method of remote sensing image [J]. Computer Engineering and Applications, 2017, 53(16): 177-181.
[14]	CUI Can, WANG Mingang, LI Li, ZHANG Ximing. Real-time compressive tracking with advanced compression features [J]. Computer Engineering and Applications, 2017, 53(15): 210-216.
[15]	ZHANG Bo1, WEN Yan2, CHEN Ming3, CHEN Tingting2. Data service aggregation algebra for supporting heterogeneous on-demand data integration [J]. Computer Engineering and Applications, 2017, 53(15): 68-76.

Power dispatch data integration model based on Spark

基于Spark的电力调度数据整合模型

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics