计算机工程与应用 ›› 2017, Vol. 53 ›› Issue (24): 80-85.DOI: 10.3778/j.issn.1002-8331.1611-0463

• 大数据与云计算 • 上一篇    下一篇

基于集群规模调整的节能存储策略

妙晓龙1,陈  浩1,钟  将2   

  1. 1.重庆大学 计算机学院,重庆 400044
    2.信息物理社会可信服务计算教育部重点实验室,重庆 400044
  • 出版日期:2017-12-15 发布日期:2018-01-09

Energy-conserving strategies of file storage based on cluster scale adjustment

MIAO Xiaolong1, CHEN Hao1, ZHONG Jiang2   

  1. 1.College of Computer Science, Chongqing University, Chongqing 400044, China
    2.Key Laboratory of Dependable Service Computing in Cyber Physical Society, Ministry of Education, Chongqing 400044, China
  • Online:2017-12-15 Published:2018-01-09

摘要: 根据谷歌数据中心研究报告,传统数据中心存在高能耗、低利用率的问题。通过研究集群数据块访问规律,提出一种基于集群规模调整的Hadoop分布式文件系统(HDFS)节能存储策略,实现HDFS高效节能存储。策略主要在集群区域划分、数据块迁移策略优化、缓存机制等方面作出了改进。实验结果表明:使用该节能策略的HDFS比传统HDFS节能35%~40%,其中0.3%的访问需要唤醒服务器,同时引入缓存策略对集群的性能提高了5.1%。

关键词: Hadoop, Hadoop分布式文件系统, 节能, 规模调整, 缓存

Abstract: According to the research report of the google data-center, traditional data-center has the problem of the high energy consumption and low utilization ratio. A Hadoop Distributed File System(HDFS) energy-saving storage scheme based on cluster scale adjustment is proposed to realize HDFS efficient energy-saving storage by the access rules of cluster data block in this paper. The strategy is improved in the cluster in the regional division, data block migration strategy optimization and caching mechanisms. The simulation experiment results show that using the energy-saving strategies of HDFS energy-saving varies between 35%~40% than traditional HDFS, 0.3% access need to wake up the servers. At the same time, while the caching storage is introducted, the performance of the cluster is improved by 5.1% compared with traditional HDFS.

Key words: Hadoop, Hadoop Distributed File System(HDFS), energy-conservation, scale adjustment, cache