计算机工程与应用 ›› 2018, Vol. 54 ›› Issue (1): 102-106.DOI: 10.3778/j.issn.1002-8331.1701-0177

• 大数据与云计算 • 上一篇    下一篇

Storm流式计算框架反压机制研究

熊安萍1,2,朱恒伟1,罗宇豪1   

  1. 1.重庆邮电大学 计算机科学与技术学院,重庆 400065
    2.重庆市移动互联网数据应用工程技术研究中心,重庆 400065
  • 出版日期:2018-01-01 发布日期:2018-01-15

Research of backpressure mechanism in Storm

XIONG Anping1,2, ZHU Hengwei1, LUO Yuhao1   

  1. 1.School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
    2.Chongqing Engineering Research Center of Mobile Internet Data Application, Chongqing 400065, China
  • Online:2018-01-01 Published:2018-01-15

摘要: Storm集群提供了强大的实时处理能力,Storm上下游处理节点由于任务差异而导致数据流元组Tuple处理超时从而影响系统吞吐量及其性能。针对该问题,提出了一种能够灵活调节Topology中各环节数据负载的反压机制,该机制采用可变队列,并根据当前Tuple负载动态调整队列大小,以适应数据负载的动态变化,并提升系统吞吐量。实验结果表明,该反压机制能够避免反压过程中出现数据流的震荡,同时提高系统性能和稳定性。

关键词: Storm集群, 反压机制, 可变队列, 数据负载, 吞吐量

Abstract: The Storm cluster provides powerful real-time processing ability. Storm node running different task in different speed results in Tuple timeout, and thus affects the system throughput and performance. This paper proposes a backpressure mechanism that can adjust data load of each section of the Topology in a flexible way. According to the load condition, the mechanism adopts variable queue to adapt to the dynamic change of the load data and improves the system throughput. The experiments show that this mechanism can avoid the shock of data flow and improve the performance and stability of Storm.

Key words: Storm cluster, backpressure, variable queue, data load, throughput