基于压缩-字对齐位图的天文海量数据实时索引

计算机工程与应用 ›› 2016, Vol. 52 ›› Issue (1): 37-41.

基于压缩-字对齐位图的天文海量数据实时索引

刘应波1，3，王锋1，2，3，季凯帆2，邓辉2，戴伟1，2，3，梁波2

1.中国科学院云南天文台，昆明 650011
2.昆明理工大学云南省计算机技术应用重点实验室，昆明 650500
3.中国科学院大学，北京 100049

出版日期:2016-01-01 发布日期:2015-12-30

Massive astronomical real-time data indexing based on compressed word-aligned bitmap

LIU Yingbo1，3, WANG Feng1，2，3, JI Kaifan2, DENG Hui2, DAI Wei1，2，3, LIANG Bo2

1.Yunnan Observatories, Chinese Academy of Sciences, Kunming 650011, China
2.Computer Technology Application Key Laboratory of Yunnan Province, Kunming University of Science and Technology, Kunming 650500, China
3.University of Chinese Academy of Sciences, Beijing 100049, China

Online:2016-01-01 Published:2015-12-30

摘要/Abstract

摘要： 澄江一米新真空大型天文望远镜（NVST）当前每天最大能产生2 TB，约十多万条的观测数据。由于这些数据量巨大并具有非结构化特性，使用离线构建索引会带来巨大时间开销，传统的关系型数据库难以满足快速索引和检索需求。针对这些问题，结合数据采集流程，提出了使用基于压缩的字对齐位图索引算法来在线实时构建索引。这种方式不仅克服了离线构建索引方式时，文件访问、FITS头读取和解析FITS头等操作带来的大量额外时间消耗问题，而且有助于解决海量太阳观测数据的高效检索难题。通过实验证明了在线实时构建索引方式能够极大地降低时间开销，也表明了该方式在天文海量数据索引和检索应用中的有效性和可行性。

关键词: 字对齐位图索引, FastBit, 海量数据, 大型望远镜

Abstract: At present, New Vacuum Solar Telescope（NVST） is generating data more than 2 TB per day, and due to the characteristic of massive non-structure data, it is beyond traditional database systems to search efficiently for a subset of these extremely large with low latency and response time and time overtime is increasing by the way of off-line index building. Aiming at these problems, combined with the strategy of on-line real data indexing in observation data capture system, using an approach of the compressed word-aligned bitmap index for observation data indexing and querying is proposed. This approach cannot only save the time cost of accessing FITS file, reading FITS file header and parsing header keyword in off-line indexing mode, but also can solve the problem of massive solar data retrieval. Experiments show that time overhead can be reduced by using real-time data indexing, and the effectiveness and feasibility of the method are proved in the experiments.

Key words: word-aligned bitmap, FastBit, massive data, large telescope

刘应波1，3，王锋1，2，3，季凯帆2，邓辉2，戴伟1，2，3，梁波2. 基于压缩-字对齐位图的天文海量数据实时索引[J]. 计算机工程与应用, 2016, 52(1): 37-41.

LIU Yingbo1，3, WANG Feng1，2，3, JI Kaifan2, DENG Hui2, DAI Wei1，2，3, LIANG Bo2. Massive astronomical real-time data indexing based on compressed word-aligned bitmap[J]. Computer Engineering and Applications, 2016, 52(1): 37-41.

[1]	杨海涛1，张传斌2，阮镇江1，徐飞1 . 大规模云同步归集数据系统的异步并行优化[J]. 计算机工程与应用, 2017, 53(2): 88-97.
[2]	王永贵，武超，戴伟. 基于MapReduce的随机抽样K-means算法[J]. 计算机工程与应用, 2016, 52(8): 74-79.
[3]	刘勇，覃飙，余萝. 海量活动轨迹相似查询[J]. 计算机工程与应用, 2015, 51(14): 99-103.
[4]	姜麟，米允龙，王添. 大数据下不完备信息系统近似空间的并行算法[J]. 计算机工程与应用, 2014, 50(15): 101-106.
[5]	冀素琴，石洪波. 面向海量数据的K-means聚类优化算法[J]. 计算机工程与应用, 2014, 50(14): 143-147.
[6]	周腾腾1，陈林祥2，胡奥1. RRTA：一种基于顺序读取的有效Top-K查询算法[J]. 计算机工程与应用, 2013, 49(17): 116-120.
[7]	燕彩蓉¹，孙圭宁²，高念高². 基于扩展树状知识库的海量数据清洗算法[J]. 计算机工程与应用, 2010, 46(28): 146-148.
[8]	傅饶，蒲静，陈华月. 面向CDN网络的高效海量数据分发机制研究[J]. 计算机工程与应用, 2010, 46(24): 126-128.
[9]	于重重^1,2,杨扬¹,涂序彦¹,张莹²,张佳². DBSCAN算法在桥梁健康监测预测模型中的应用[J]. 计算机工程与应用, 2008, 44(12): 224-227.