关键词: 高能物理, 数据管理, 数据传输, 性能优化, 多流传输

Abstract: High-energy physics experiments continue to progress and develop, they have produced PB even EB-level of data. The data acquisition, storage, transmission, sharing, analysis and management are facing great problems and challenges. In order to meet these challenges, this paper designs and implements an event-oriented data management system, which effectively solves the problem of low efficiency of data processing and low utilization of resources. Firstly, an indexing system based on Nosql database is designed. By extracting specified properties of events, the most interesting attributes of the physicists are selected and are stored in the database. The inverted indexing technology is adopted to improve the efficiency of event retrieval. Then, it caches the sample data for optimization, reducing data conversion and storage overhead. Also, the data cross-domain transmission scheme is proposed, which makes full use of the network bandwidth and reduces the delay of processing the data. The data transfer system based on a web framework and asynchronous networking library is called tornado. By using non-blocking network I/O, Tornado can scale to tens of thousands of open connections, making it ideal for long polling. The experimental results show that the event-level indexing technology can significantly improve the retrieval efficiency of the event data , and the network bandwidth of the data transmission system can be utilized more than 90%.

Key words: high energy physics, data management, data transfer, performance optimization, multi-flow transmission