Research on vector spatial data distributed computing using Hadoop projects

Computer Engineering and Applications ›› 2013, Vol. 49 ›› Issue (16): 25-29.

Previous Articles Next Articles

Research on vector spatial data distributed computing using Hadoop projects

YIN Fang1, FENG Min2, ZHU Yunqiang2, LIU Rui3

1.College of Earth Science and Resources，Chang’an University，Xi’an 710054，China
2.Institute of Geographic Sciences and Natural Resources Research，Chinese Academy of Sciences，Beijing 100101，China
3.College of Geographical Science，Chongqing Normal University，Chongqing 400047，China

Online:2013-08-15 Published:2013-08-15

基于开源Hadoop的矢量空间数据分布式处理研究

尹芳1，冯敏2，诸云强2，刘睿3

1.长安大学地球科学与资源学院，西安 710054
2.中国科学院地理科学与资源研究所，北京 100101
3.重庆师范大学地理科学学院，重庆 400047

Abstract

Abstract: The paper designs a vector spatial data distributed computing system based on Open Source Hadoop Projects, in order to satisfy the needs of massive vector data. According to the characteristics of the vector spatial data, Key/Value data model and GeoJSON data format, the paper brings forward a distributed Key/Value storage method for vector spatial data based on HDFS. The key techniques on how to computing large-scale vector spatial data based on MapReduce are elaborated in detail, including data partitioning and parallel processing mechanism of Map step, results merging of Reduce step. A vector spatial data distributed computing prototype system is developed using Open Source Hadoop projects and applied to deal with the 1∶100, 000 land use data of Guanzhong area in China. The evaluation result indicates that the Hadoop MapReduce can significantly leverage the performance of vector spatial data analysis, especially when more computing nodes are used.

Key words: vector spatial data, Key/Value, GeoJSON, Apache Hadoop, MapReduce, distributed computing

摘要： 为实现大规模矢量数据的高性能处理，在开源项目Hadoop基础上，设计与开发了一个基于MapReduce的矢量数据分布式计算系统。根据矢量空间数据的特点，通过分析Key/Value数据模型及GeoJSON地理数据编码格式，构建了可存储于Hadoop hdfs的矢量数据Key/Value文本文件格式；探讨矢量数据的MapReduce计算过程，对Map数据分片、并行处理过程及Reduce结果合并等关键步骤进行了详细阐述；基于上述技术，建立了矢量数据分布式计算原型系统，详细介绍系统组成，并将其应用于处理关中地区1∶10万土地利用矢量空间数据，取得较好效果。

关键词: 矢量空间数据, Key/Value, GeoJSON, Apache Hadoop, MapReduce, 分布式处理

YIN Fang1, FENG Min2, ZHU Yunqiang2, LIU Rui3. Research on vector spatial data distributed computing using Hadoop projects[J]. Computer Engineering and Applications, 2013, 49(16): 25-29.

尹芳1，冯敏2，诸云强2，刘睿3. 基于开源Hadoop的矢量空间数据分布式处理研究[J]. 计算机工程与应用, 2013, 49(16): 25-29.

[1]	CHEN Yuanwen. Application of MapReduce Technology in Problem of Material Transportation and Stowage [J]. Computer Engineering and Applications, 2021, 57(12): 273-278.
[2]	LIU Jun, LI Wei, WU Mengting, CHEN Qifeng. New Design of Image Parallel Processing Model Based on Hadoop Platform [J]. Computer Engineering and Applications, 2019, 55(6): 186-190.
[3]	JI Changqing1，2, XIAO Peng3, LIU Chang4, WANG Zumin2, XI Fang2, SHAO Yinbo1, LI Zeyu2. Mobile Medical Call Algorithms Based on Spatial kNN Query [J]. Computer Engineering and Applications, 2019, 55(2): 206-212.
[4]	LIU Kang1, ZHANG Xueying1, LI Fenglian1, TIAN Yuchu1，2. Graph partitioning method for social networks based on communication load balancing [J]. Computer Engineering and Applications, 2018, 54(4): 66-71.
[5]	HU Deqi, SUN Yongqi, QIN Chao. Application of L1-BC algorithm based on Spark in key proteins detection [J]. Computer Engineering and Applications, 2018, 54(24): 234-240.
[6]	SONG Dongfei, XU Hua. Research and parallelization of DBSCAN algorithm [J]. Computer Engineering and Applications, 2018, 54(24): 52-56.
[7]	WANG Dezheng1, ZHANG Yinong1, YANG Fan2. Implementation of parallel PLS algorithm of process monitoring using MapReduce [J]. Computer Engineering and Applications, 2018, 54(24): 61-65.
[8]	LI Gefei1, MA Weiyin2, LI Li3. Convex hull problem based on Spark platform [J]. Computer Engineering and Applications, 2018, 54(22): 67-73.
[9]	CHEN Wanghu, YU Maoyi, MA Shengjun. Training BP neural networks with MapReduce based on sample data slice disruptions [J]. Computer Engineering and Applications, 2018, 54(2): 137-143.
[10]	XIA Xiaoyun, ZHANG Renbin, XIE Rui, WANG Cong. MapReduce approach for defect inspection of TFT-LCD [J]. Computer Engineering and Applications, 2017, 53(5): 202-206.
[11]	WANG Yonggui1, ZHANG Yan1, YANG Dongdong2. Research on algorithm of community discovery of wireless city based on MapReduce [J]. Computer Engineering and Applications, 2017, 53(4): 106-112.
[12]	LUO Jun, LI Jinhua. LSHBMRPK-means algorithm and its application. Computer Engineering and Applications [J]. Computer Engineering and Applications, 2017, 53(21): 62-67.
[13]	CHEN Yanan, ZHU Xijun. Association analysis of TCM asthma medication combination based on Hadoop [J]. Computer Engineering and Applications, 2017, 53(13): 95-98.
[14]	LI Sanmiao, LI Longshu. Performance analysis of four methods for handling small files in Hadoop [J]. Computer Engineering and Applications, 2016, 52(9): 44-49.
[15]	XIONG Zheng1，2, WANG Jinming2, ZHENG Haiyan1，2, LI Kunming1, XU Lizhen2, CHONG Zhihong2. RDF pattern matching algorithm with sorted view [J]. Computer Engineering and Applications, 2016, 52(8): 62-69.

Research on vector spatial data distributed computing using Hadoop projects

基于开源Hadoop的矢量空间数据分布式处理研究

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics