微服务架构磁带库存储系统设计与实现

doi:10.3778/j.issn.1002-8331.2206-0067

摘要/Abstract

摘要： 建立具有磁带存储层的HDFS分级存储系统是完善高能物理领域Hadoop生态系统的重要部分，但高能物理领域传统的磁带存储管理系统（如Castor、CTA）上层不支持HDFS磁盘存储，并且随着高能物理数据量的急剧增长、互联网技术的不断发展和用户需求的迅猛变化，传统的磁带存储管理系统逐渐呈现出系统扩展、负载均衡、开发和运维成本上升等方面的问题。设计开发了基于微服务架构的磁带存储管理系统。该系统向上支持HDFS磁盘存储，将磁带库资源管理、文件传输、磁带读写等功能以微服务的形式分布到各个服务实例中，达到分散服务压力的目的，并且系统针对传统负载算法效率不佳的问题，实现了基于服务器响应指数的负载均衡算法，通过根据自定义参数计算得到的服务器响应指数对其进行排序，保证将用户请求调度到响应指数最高的服务器进行处理。实验结果表明，磁带库存储系统满足HDFS文件分级存储磁带层管理的需求，提出的基于服务器响应指数的负载均衡算法相较于轮询算法，系统归档性能高出6%以上，提取性能高出64%以上；相较于随机算法，系统归档性能高出9%以上，提取性能高出64%以上，最终实现的磁带库存储系统表明，与传统的系统相比，微服务体系结构能够实现系统中组件的解耦和分布式负载的平衡，在系统开发和运维等方面更为便捷。

关键词: 磁带库存储, 微服务架构, 负载均衡, 响应指数计算模型, HDFS系统

Abstract: The establishment of HDFS tiered storage system with tape storage layer is an important part of improving Hadoop ecosystem in the field of high energy physics（HEP）. However, the traditional tape storage management system in HEP（such as Castor and CTA） doesn’t support HDFS. In addition, with the rapid growth of the amount of HEP data, the continuous development of current Internet technology and rapid changes in user needs, the traditional tape storage management system gradually presents the problem such as system expansion, load balancing, development and maintenance cost increases sharply. This paper designs and develops a tape storage management system based on microservice architecture. The system supports HDFS disk storage and distributes tape library resource management, file transfer, tape read/write and so on functions to each service instance in the form of microservices to disperse service pressure. Furthermore, aiming at the problem of low efficiency of traditional load algorithm, the system proposes and implements a load balancing algorithm based on server response factor. It sorts severs according to the server response factor calculated by user-defined parameters to ensure that user requests are scheduled to the server with the highest response factor for processing. ?Experimental results show that the tape library storage system in this paper can meet the requirements of HDFS file hierarchical storage. Compared with polling algorithm, the system performance of archiving based on server response factor is more than 6% better, and that of extraction is more than 64% better. ?Compared with random algorithm, archiving performance is more than 9% better, and extraction performance is more than 64% better. The application results of the system indicates that compared with the traditional system, the microservice architecture can decouple components and balance distributed loads in the system, and is more convenient in system development and maintenance.

Key words: tape library storage, microservice architecture, load balancing, response index calculation model, HDFS system

刘晓宇, 夏立斌, 姜晓巍, 孙功星. 微服务架构磁带库存储系统设计与实现[J]. 计算机工程与应用, 2023, 59(15): 253-263.

LIU Xiaoyu, XIA Libin, JIANG Xiaowei, SUN Gongxing. Design and Implementation of Tape Library Storage System Based on Microservice Architecture[J]. Computer Engineering and Applications, 2023, 59(15): 253-263.

参考文献

[1] HUANG Q L，WEI Z C，SUN G X，et al.Using hadoop for high energy physics data analysis[C]//International Conference on Big Scientific Data Management（BigSDM 2018），2018.
[2] SHVACHKO K，KUANG H，RADIA S，et al.The hadoop distributed file system[C]//26th Symposium on Mass Storage Systems and Technologies（MSST），2010.
[3] 尹乔，魏占辰，黄秋兰，等.Hadoop海量数据迁移系统开发及应用[J].计算机工程与应用，2019，55（13）：66-71.
YIN Q，WEI Z C，HUANG Q L，et al.Development and application of Hadoop massive data migration system[J].Computer Engineering and Applications，2019，55（13）：66-71.
[4] 陈刚.高能物理实验中的数据与计算技术[J].中国科学：物理学、力学、天文学，2021，51（9）：14-23.
CHEN G.Data and computing for high energy physics experiments[J].Scientia Sinica：Physica，Mechanica & Astronomica，2021，51（9）：14-23.
[5] 程耀东，石京燕，陈刚.高能物理计算环境概述[J].科研信息化技术与应用，2014，5（3）：3-10.
CHENG Y D，SHI J Y，CHEN G.Overview of high energy physics computing environment[J].Information Technology and Application of Scientific Research，2014，5（3）：3-10.
[6] DRAGONI N.et al.Microservices：yesterday，today，and tomorrow[M].Cham：Spring，2017：195-216.
[7] FOWLER M，LEWIS J.Microservices[EB/OL].[2022-03-03].https：//martinfowler.com.
[8] LO PRESTI G，BARRING O，EARL A，et al.CASTOR：a distributed storage resource facility for high performance data processing at CERN[C]//24th IEEE Conference on Mass Storage Systems and Technologies（MSST 2007）.San Diego，CA，USA：IEEE，2007.
[9] CANO E，BAHYL V，CAFFY C，et al.CERN tape archive：production status，migration from CASTOR and new features[C]//24th International Conference on Computing in High Energy and Nuclear Physics（CHEP 2019）.Adelaide，Australia：EDP Sciences，2020.
[10] DORIGO A，ELMER P，FURANO F，et al.XROOTD-a high scalable architecture for data access[J].WSEAS Transactions on Computers，2005，1：348-353.
[11] AYDIN S，AYDIN M N.Design and implementation of a smart beehive and its monitoring system using microservices in the context of IoT and open data[J].Computers and Electronics in Agriculture，2022，196：106897.
[12] MATLEKOVIC L，JURIC F，SCHNEIDER-KAMP P.Microservices for autonomous UAV inspection with UAV simulation as a service[J].Simulation Modelling Practice and Theory，2022，119：102548.
[13] BAI D X，TANG J T，LU G Y，et al.The design and application of landslide monitoring and early warning system based on microservice architecture[J].Geomatics，Natural Hazards and Risk，2020，11（1）：928-948.
[14] YAN L，CAO S Z，GONG Y S，et al.SatEC：a 5G satellite edge computing framework based on microservice architecture[J].Sensors，2019，19（4）：831.
[15] KHOSO F H，LAKHAN A，ARAIN A A，et al.A microservice-based system for industrial internet of things in fog-cloud assisted network[J].Engineering，Technology & Applied Science Research，2021，11（2）：7029-7032.
[16] LYU Z L，WEI H，BAI X Q，et al.Microservice-based architecture for an energy management system[J].IEEE Systems Journal，2020，14（4）：5061-5072.
[17] LIAN J，GAO M.Design and implementation of network information service platform based on microservice architecture[C]//2020 3rd International Conference on Mechatronics and Computer Technology Engineering，2020.
[18] 邵瑛，徐斌.基于微服务的通用性系统日志评估方案及应用[J].计算机系统应用，2022，31（1）：124-131.
SHAO Y，XU B.Log evaluation scheme and application of general system based on microservice[J].Computer System & Applications，2022，31（1）：124-131.
[19] 郇凯，黄佳为，王鑫，等.基于微服务架构的管道地质灾害监测预警系统[J].计算机系统应用，2022，31（3）：65-74.
HUAN K，HUANG J W，WANG X，et al.Pipeline geological disaster monitoring and early warning system based on micro-service architecture[J].Computer Systems & Application，2022，31（3）：65-74.
[20] 骆光磊，宋云涛.基于微服务架构的航道工程BIM综合管理平台技术研究[J].中国港湾建设，2021，41（12）：75-79.
LUO G L，SONG Y T.Technical research on BIM integrated management platform of waterway engineering based on micro service architecture[J].China Harbour Engineering，2021，41（12）：75-79.
[21] AL-DEBAGY O，MARTINEK P.A comparative review of microservices and monolithic architectures[C]//International Symposium on Computational Intelligence and Informatics.Budapest，Hungary：IEEE Press，2018.
[22] InfoQ.From a monolith to microservices+REST：the evolution of LinkedIn’s service architecture[EB/OL].[2022-03-04].https：//www.infoq.com/presentations/linkedin-microservices-urn.
[23] JAQUETTE G A.LTO：a better format for mid-range tape[J].IBM Journal of Research and Development，2003，47（4）：429-444.
[24] PAMULA N B，JAIRAM K，RAJESH B.Cache-aside approach for cloud design pattern[J].International Journal of Computer Science and Information Technologies，2014，5（2）：1423-1426.
[25] ZHENG Q，YANG T，KAN Y，et al.On the analysis of cache invalidation with LRU replacement[J].IEEE Transactions on Parallel and Distributed Systems，2021，33（3）：654-666.
[26] WU Z J，DAI H，YUAN Y K，et al.Intelligent blockchain system based on differential privacy[J].Journal of Physics：Conference Series，2022，15（1）：012009.
[27] LIU Y，ECKERT C M，EARL C.A review of fuzzy AHP methods for decision-making with subjective judgements[J].Expert Systems with Applications，2020，161：113738.
[28] 奚智雯，蔡晶晶，阳文敏，等.微服务架构FPGA云平台的并发请求调度机制研究[J].计算机工程，2022，48（7）：206-213.
XI Z W，CAI J J，YANG W M，et al.A study of concurrent request scheduling mechanism for an FPGA cloud based on microservice architecture[J].Computer Engineering，2022，48（7）：206-213.
[29] 杨善超，田康生，李宏权，等.综合优先级下反导预警相控阵雷达任务调度算法[J].兵工学报，2020，41（2）：315-323.
YANG S C，TIAN K S，LI H Q，et al.Comprehensive priority-based task scheduling algorithm for anti-missile early warning phased array radar[J].Acta Armamentarii，2020，41（2）：315-323.
[30] 董君.层次分析法权重计算方法分析及其应用研究[J].科技资讯，2015，13（29）：218-220.
DONG J.Analytic hierarchy process weight calculation method analysis and its application research[J].Science and Technology Information，2015，13（29）：218-220.
[31] 刘卫锋，何霞.基于模糊判断矩阵的两阶段群决策方法[J].计算机工程，2012，38（10）：141-143.
LIU W F，HE X.Two-stage group decision making method based on fuzzy judgment matrix[J].Computer Engineering，2012，38（10）：141-143.
[32] RAJ V，SADAM R.Performance and complexity comparison of service oriented architecture and microservices architecture[J].Communication Networks and Distributed Systems，2021，27（1）：100-117.