一种改进的基于缓存池机制的小文件I/O策略

计算机工程与应用 ›› 2016, Vol. 52 ›› Issue (16): 210-215.

一种改进的基于缓存池机制的小文件I/O策略

聂瑞华1，谢文君1，梁军2

1.华南师范大学计算机学院，广州 510631
2.华南师范大学网络中心，广州 510631

出版日期:2016-08-15 发布日期:2016-08-12

Improved I/O policy of small files based on cache pool

NIE Ruihua1, XIE Wenjun1, LIANG Jun2

1.School of Computer, South China Normal University, Guangzhou 510631, China
2.Network Center, South China Normal University, Guangzhou 510631, China

Online:2016-08-15 Published:2016-08-12

摘要/Abstract

摘要： Lustre文件系统对大文件的I/O性能较好，但对小文件不佳。针对这个问题，提出建立一个基于MDS节点的小文件缓存池机制，在缓存池里缓存经常被访问的小文件。在该机制中，小文件缓存池与OST使用全相联映射方式对应，并且使用贯穿读出式和直写式策略保持文件的一致性；缓存池更新策略综合考虑了文件的访问时间和次数等因素，使用改进的近期最少使用算法（LRU）更新替换。实验结果表明，改进后的Lustre文件系统减少了小文件的网络传输开销和访问时间，对小文件的I/O性能有较明显的提高。虽然它对大文件的I/O性能有所降低，但在可接受范围之内，仍具有一定的实用价值。

关键词: Lustre文件系统, 全相联映射, 近期最少使用算法（LRU）, I/O性能

Abstract: The I/O performance is excellent for the large files on Lustre file system, but is poor for the small files. To solve the problem, this paper proposes a small file cache pool mechanism based on MDS node, the frequently accessed small files are put in the pool. In this mechanism, the pool uses a fully associative mapping to correspond with OST file cache, and uses look through style and write through strategy to ensure the file consistency. The cache pool updating strategy considers some factors that include the file access time and number, and uses a modified Least Recently Used algorithm（LRU）. Experimental results show that the improved Lustre file system reduces the network transmission overhead and saves access time of small files, and I/O performance has been significantly improved on accessing small files. Although I/O performance has been reduced on accessing large files, the improved Lustre file system still has some practical value within an acceptable range.

Key words: Lustre file system, fully associative mapping, Least Recently Used（LRU）, I/O performance

聂瑞华1，谢文君1，梁军2. 一种改进的基于缓存池机制的小文件I/O策略[J]. 计算机工程与应用, 2016, 52(16): 210-215.

NIE Ruihua1, XIE Wenjun1, LIANG Jun2. Improved I/O policy of small files based on cache pool[J]. Computer Engineering and Applications, 2016, 52(16): 210-215.