关键词: MapReduce工作负载, 优化, 数据流依赖, 资源依赖, 工作流优化器

Abstract: Optimizations at different levels of the MapReduce stack have their advantages and disadvantages. For MapReduce workload optimization problem, related concepts are proposed; cost-based optimization approaches and related technology of MapReduce jobs are introduced and assessed through comparation with RoT; three MapReduce workflow optimizers are presented for cost-based optimization of MapReduce workflows based on dataflow and resource dependencies. Cost-based workflow optimization is evaluated. End-to-end evaluation of the workflow optimizer is described; the advantages and disadvantages of these three workflow optimizers are analyzed through experimental evaluation of their overhead.

Key words: MapReduce workloads, optimization, dataflow dependencies, resource dependencies, workflow optimizer