计算机工程与应用 ›› 2017, Vol. 53 ›› Issue (9): 80-84.DOI: 10.3778/j.issn.1002-8331.1601-0040

• 大数据与云计算 • 上一篇    下一篇

基于HBase的气象结构化数据查询优化

徐熙超1,2,杨  铮1,马廷淮1,2   

  1. 1.南京信息工程大学 计算机与软件学院,南京 210044
    2.南京信息工程大学 江苏省网络监控中心,南京 210044
  • 出版日期:2017-05-01 发布日期:2017-05-15

Optimization of meteorological structured data based on HBase

XU Xichao1,2, YANG Zheng1, MA Tinghuai1,2   

  1. 1.College of Computer and Software, Nanjing University of Information Science and Technology, Nanjing 210044, China
    2.Jiangsu Engineering Center of Network Monitoring, Nanjing University of Information Science and Technology, Nanjing 210044, China
  • Online:2017-05-01 Published:2017-05-15

摘要: 海量气象观、探测数据是提高公共气象服务精细化、精准化和个性化水平的关键。日增TB级的海量气象数据在存储、检索、传输、共享方面的时效性要求对构建在传统的IOE技术架构上气象数据管理系统提出了严峻挑战。在HBase基础上,提出了一个基于索引的气象结构化数据查询优化架构HBase4M(HBase for Meteorology)。首先,根据HBase存储特性设计表结构;然后,利用协处理器建立和维护辅助索引,将字段查询转化为对索引表的行键查询,使得HBase4M在具备HBase可扩展性、低延迟的特性上可以支持结构化气象数据的灵活查询。实验结果表明,HBase4M的性能可以基本满足气象服务的业务需要。

关键词: 气象数据, HBase, 辅助索引, 分布式存储

Abstract: Massive meteorological observations data is the key of the improvement of public weather services’ refinement, precision and personalized level. The increased meteorological data can reach to 1 TB every day. The timeless requirements of large scale meteorological data for its storage, retrieval, transfer and share is a challenge to traditional meteorological information system while based on IOE(IBM, Oracle and EMC)framework. Based on HBase, this paper proposes a structural query optimization using index, named as Hbase4M(HBase for Meteorology). Firstly, the table structure is designed according to the HBase storage property. Secondly, the secondary indexes are established and maintained by HBase coprocessors. It will convert the field query to rowkey search in index table. It also facilitates HBase4M directly to inherit HBase’s scalability and low latency and support the flexibility of structure meteorological data query. Experimental results show that the performance of HBase4M can satisfy the demands of meteorological business services.