关键词: Flink, MongoDB, Flink与MongoDB连接中间件, 数据逻辑分片, 并行读写

Abstract: In order to improve the reading and writing rate between big data processing platform Flink and MongoDB, this paper proposes and implements an efficient connection middleware of Flink and MongoDB. Based on Flink’s parallelization idea, by logically fragmenting the data, the interface in the Mongo-Java package is called to realize parallel reading and writing of data. With different scale of hydrological sensor datasets as experimental data, the reading and writing speeds of the data in Java single-threaded operation, Hadoop and MongoDB connector and the Flink and MongoDB connection middleware proposed in this paper are tested. The results show that the efficiency of using Flink to read and write data is 1.5 times higher than the single-threaded operation, which validates that the connection middleware can effectively improve the reading and writing speed of massive data.

Key words: Flink, MongoDB, Flink-MongoDB connector middleware, logically fragmenting data, parallel reading and writing