Computer Engineering and Applications ›› 2016, Vol. 52 ›› Issue (11): 248-253.

Previous Articles     Next Articles

Research of distributed storage method for agricultural scientific data

WANG Jian1, HUANG Chaoguang2, WANG Jian1, LIU Shaokun2, CHAI Jin2   

  1. 1.Agricultural Information Institute, Chinese Academy of Agricultural Sciences, Beijing 100081, China
    2.College of Computer Science and Technology, Beijing University of Technology, Beijing 100022, China
  • Online:2016-06-01 Published:2016-06-14

面向农业科学数据的分布式存储方法研究

王  剑1,黄朝光2,王  健1,刘少坤2,柴  进2   

  1. 1.中国农业科学院 农业信息研究所,北京 100081
    2.北京工业大学 计算机学院,北京 100022

Abstract: With the further development of agriculture technology, the amount of data of agriculture science is crazy geometric growth. To face the rapid growth of agriculture data, how to store and manage vast amounts of agriculture data effectively has become a research hotspot. In this passage, a distributed storage method is presented for agricultural data by means of the advantages of Hadoop framework. Combined with the technology of message communication and the strategy of hybrid index distribution, the method that adopts “central control node-the node data” storage system is proved to be efficient for mass data storage and retrieval. Experimental results show that the proposed method of distributed storage can be applicable to most types of the vast amounts of data storage and the effective manage for data can be achieved.

Key words: distributed storage, agricultural scientific data, Hadoop, concurrent access, MapReduce

摘要: 随着农业科技的飞速发展,农业科学数据以几何级数快速膨胀。面对持续增长的农业数据资源,如何有效地存储和管理海量的农业数据成为一个研究热点。借助Hadoop分布式存储框架的优势,提出了一种面向农业科学数据的分布式存储方法,该方法采用了“中心控制节点——数据节点”的存储体系,通过报文通信技术和混合式索引分布策略,实现了对海量数据的高并发式存储和检索。实验结果证明,该方法适用于各种类型的农业科学数据进行存储管理。

关键词: 分布式存储, 农业科学数据, Hadoop, 并发访问, MapReduce