关键词: 异步并行处理, 海量数据归集, 大规模云同步, 数据同步枢纽

Abstract: To establish data centers for non-vertically administrated industries or spheres, a software system capable of sync-collecting industrial data in large-scale is needed to be equipped. To this need, the so-called Generic Sync-pivot Mega-data Syndicate platform for industries data clouds, in short GSMS is developed. GSMS is mainly used to sync-collect massive business or factual data from heterogeneous autonomous systems or equipments, and thereon develop and apply those data. In practical applications, when each of numerous GSMS client threads concurrently sync data into a GSMS data center respectively, they make a convergence of large-scale sync sessions at the GSMS server end, thereby it forms a process bottleneck. Besides, the lock-step mechanism which makes the entire course of sync session serial will depress performance of large-scale data sync-collection. Thus, a scheme of asynchronous paralleling improvement upon GSMS is proposed and developed：separating highly time-consuming computing step from data sync lock-step course, and introducing therein an asynchronous paralleling mechanism based on multiple message queues middleware, as well as providing corresponding measures for committing loose transaction of data sync. Practices demonstrate that the right implementation of such asynchronous paralleling can efficiently upgrade processing speed at the server end while still satisfying reliability and consistency requirements of sync system.

Key words: asynchronous paralleling, massive data collection, large-scale cloud sync, data sync pivot