Quick-View Processing Method for Remote Sensing Data Based on Stream Computing

doi:10.3778/j.issn.1002-8331.1801-0342

Computer Engineering and Applications ›› 2019, Vol. 55 ›› Issue (10): 77-82.DOI: 10.3778/j.issn.1002-8331.1801-0342

Previous Articles Next Articles

Quick-View Processing Method for Remote Sensing Data Based on Stream Computing

SONG Yao1，2，3, SUN Xiaojuan1，2，3, HU Yuxin1，2，3, LEI Bin1，2，3, LU Xiaojun4

1.University of Chinese Academy of Sciences, Beijing 100049, China
2.Institute of Electronics, Chinese Academy of Sciences, Beijing 100190, China
3.Key Laboratory of Technology in Geo-spatial Information Processing and Application System, Chinese Academy of Sciences, Beijing 100190, China
4.China International Engineering Consulting Corporation, Beijing 100048, China

Online:2019-05-15 Published:2019-05-13

基于流式计算的遥感卫星数据快视处理方法

宋峣1，2，3，孙小涓1，2，3，胡玉新1，2，3，雷斌1，2，3，卢晓军4

1.中国科学院大学，北京 100049
2.中国科学院电子学研究所，北京 100190
3.中国科学院空间信息与应用系统重点实验室，北京 100190
4.中国国际工程咨询公司，北京 100048

Abstract

Abstract: With the improvement of data acquiring ability of high resolution remote sensing satellites and data receiving ability of ground receiving stations, the processing load of existing system grows increasingly heavier and the real-time processing demand becomes more difficult to meet. Focusing on these issues, this paper proposes a new system design method for quick-view processing of remote sensing satellite data by using the thought of stream computing. After analyzing the characteristics of quick-view processing data streams, this paper applies the Storm framework to the parallel optimization of the existing system, designs the topology of stream computing tasks for the remote sensing satellite data processing, and uses Kafka message oriented middleware to improve the mechanism of data exchanging and data buffering in processing units. In experiments the improved system shows good results in throughput and reliability.

Key words: stream computing, data stream, Storm, quick-view processing, remote sensing data processing

摘要： 随着高分辨率遥感卫星数据获取能力和地面数传接收能力的提高，现有遥感卫星快视处理系统的处理负载增大，实时性要求越来越难以满足。针对这些问题，采用流式计算思想提出了一种新的遥感卫星数据快视处理系统设计方法。在分析遥感卫星数据快视处理数据流特点的基础上，应用Storm框架对现有系统进行并行优化，设计遥感数据流处理任务拓扑结构，同时利用消息队列中间件Kafka改进处理单元间数据交换和数据缓存方式。实验表明，该系统在数据吞吐率和可靠性方面测试效果良好。

关键词: 流式计算, 数据流, Storm, 快视处理, 遥感数据处理

SONG Yao1，2，3, SUN Xiaojuan1，2，3, HU Yuxin1，2，3, LEI Bin1，2，3, LU Xiaojun4. Quick-View Processing Method for Remote Sensing Data Based on Stream Computing[J]. Computer Engineering and Applications, 2019, 55(10): 77-82.

宋峣1，2，3，孙小涓1，2，3，胡玉新1，2，3，雷斌1，2，3，卢晓军4. 基于流式计算的遥感卫星数据快视处理方法[J]. 计算机工程与应用, 2019, 55(10): 77-82.

[1]	WANG Junhong, GUO Yahui. Imbalanced Data Stream Classification Algorithm for Dynamic Data Chunk [J]. Computer Engineering and Applications, 2021, 57(13): 124-129.
[2]	ZHOU Yu, ZHU Wenhao, FANG Qian, BAI Lei. Survey of Outlier Detection Methods Based on Clustering [J]. Computer Engineering and Applications, 2021, 57(12): 37-45.
[3]	XU Qingyan, HE Li, ZHU Hongxi. Improved Detection Method of Concept Drift Based on the Hoeffding Inequality [J]. Computer Engineering and Applications, 2020, 56(19): 55-61.
[4]	HU Yang, HU Xuegang, LI Peipei. Fast Short Text Data Stream Classification Method Based on Spark [J]. Computer Engineering and Applications, 2020, 56(14): 138-147.
[5]	YANG Hang, ZHU Yongli. Ensemble Empirical Mode Decomposition of Partial Discharge Signal Based on Storm [J]. Computer Engineering and Applications, 2020, 56(10): 261-267.
[6]	CONG Yihao, YU Yanhua. Online Distributed Recommendation Algorithm Based on Data Stream and Peer-to-Peer Network [J]. Computer Engineering and Applications, 2019, 55(1): 64-69.
[7]	JIANG Yi1，2, LUO Yuhao1, ZHU Hengwei1. Research of scheduling based on Topology under Storm clusters [J]. Computer Engineering and Applications, 2018, 54(7): 84-88.
[8]	WEI Zijin1，2，3, XIAO Li2，3. Parallel out-of-core model simplification algorithm based on improved vertex clustering [J]. Computer Engineering and Applications, 2018, 54(13): 181-190.
[9]	LIN Shijie1，2, DONG Chen1，2, CHEN Mingzhi1，2, ZHANG Fan1，2, CHEN Jinghui1，2. Summary of new group intelligent optimization algorithms [J]. Computer Engineering and Applications, 2018, 54(12): 1-9.
[10]	XIONG Anping1，2, ZHU Hengwei1, LUO Yuhao1. Research of backpressure mechanism in Storm [J]. Computer Engineering and Applications, 2018, 54(1): 102-106.
[11]	KONG Xiangxuan1, LIAO Husheng2, WANG Yixiong1, GAO Hongyu1, SU Hang1. Distributed complex event stream processing platform research [J]. Computer Engineering and Applications, 2017, 53(8): 68-75.
[12]	LIU Xiao, LIU Huiping, JIN Cheqing. Approximate solution for ER-Topk query upon uncertain data stream [J]. Computer Engineering and Applications, 2017, 53(4): 98-105.
[13]	HAN Chong1, YUAN Yingshan2, MEI Tao2, GENG Huiling2. Data stream outlier detection algorithm based on K-means [J]. Computer Engineering and Applications, 2017, 53(3): 58-63.
[14]	SHI Yingzhong 1, 2, CAO Jianfeng2, DENG Zhaohong1, JIANG Yizhang1. Flexible drift support vector machines for data stream classification [J]. Computer Engineering and Applications, 2017, 53(23): 118-122.
[15]	ZHOU Zhiyang, FENG Baiming, YANG Penglin, WEN Xianghui. Research and Implementation of KNN classification algorithm for streaming data based on Storm [J]. Computer Engineering and Applications, 2017, 53(19): 71-75.

Quick-View Processing Method for Remote Sensing Data Based on Stream Computing

基于流式计算的遥感卫星数据快视处理方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics