关键词: Alluxio分布式内存存储系统, 数据随机访问, 内存计算, 内存映射, 科学计算

Abstract:

Alluxio simplifies the implementation of file system to pursue scalability and fault tolerance, like not supporting data random access, which leads it cannot be available to many applications. Also Alluxio’s native Java interface is not flexible to support traditional programs and is not able to fully use the high-speed performance of memory. In order to address these issues, a new method to support data random access is proposed. The new method allows the tasks to random read/write data on the local file system as well local memory instead of using original Alluxio data streaming interface. With random access to local memory, this work can greatly improve the read/write performance. Test results show that write performance can be improved by 14.5% and read performance can be increased by more than 1.4 times compared to the native Java interface. Combined with application, this new method can achieve up to tens times of performance improvement.

Key words: Alluxio distributed in-memory storage system, data random access, memory computing, memory mapping, scientific computing