关键词: 日志数据, 框架, 收集处理, 分布式收集, 多层次数据存储, 查询服务

Abstract: As modern computer systems continue to grow in scale and complexity, the number of its running log is also increasing. Due to the huge size of the log data, and the content is messy, the user has some difficulties in viewing the log data. Therefore, it is of great significance to effectively collect and process the log data of application running log. This paper proposes a framework for collection and service processing of application running log, using distributed collection strategy to collect log data, defining a multi-level data storage structure to store log data, and providing the user with log data query services. It analyzes the log data of an integrated disaster reduction system. It is used to verify the practicability of the proposed framework, and it is compared with the traditional log collection and analysis software ELK and Chukwa.

Key words: log data, framework, collection and processing, distributed collection, multi-level data storage, query service