关键词: Hadoop, Map/Reduce, 监控, 数据访问

Abstract: Aiming on the issue of task scheduler considering the data location information for locality-based data processing in Hadoop Map tasks, a novel data access behavior monitoring mechanism is proposed in this paper. It is argued that the data access monitoring mechanism of Hadoop platform should not only serve to promote the efficiency of data access, but also serve to promote the execution efficiency of parallel Map/Reduce jobs. It is necessary to monitor the balance of data access overhead in the parallel execution of multiple Map tasks. The granularity and information set of data access monitoring in Hadoop platform is defined; The master-slave-based monitoring architecture is presented, which works with the support of Hadoop existing function modules; The detail implementation of the main monitoring function modules is discussed and the experimental results is analyzed.

Key words: Hadoop, Map/Reduce, monitoring, data access