Decentralized Federated Learning Strategy for Non-Independent and Identically Distributed Data

doi:10.3778/j.issn.1002-8331.2204-0165

Abstract

Abstract: Blockchain has the characteristics of immutability and decentralization, and its combination with federated learning has become a hot topic in the field of artificial intelligence. At present, decentralized federated learning has the problem of performance degradation caused by non-IID distribution of training data. In order to solve this problem, the paper firstly proposes a calculation method of model similarity, and then designs a decentralized federated learning strategy based on the similarity of the model. The learning strategy is tested using five federated learning tasks, using the CNN model to train the fashion-mnist dataset, the alexnet model to train the cifar10 dataset, the TextRnn model to train the THUsnews dataset, the Resnet18 model to train the SVHN dataset and the LSTM model to train the sentiment140 dataset. The experimental results show that the designed strategy performs decentralized federated learning under the non-IID data of the five tasks, and the accuracy rates are increased by 2.51, 5.16, 17.58, 2.46 and 5.23?percentage points respectively.

Key words: blockchain;federated learning, deep learning, data privacy

摘要： 区块链具有不可篡改性和去中心化的特点，其与联邦学习的结合成为人工智能领域的热门主题。目前去中心化联邦学习存在训练数据非独立同分布导致的性能下降问题，为了解决这个问题，提出一种模型相似度的计算方法，然后设计一种基于该模型相似度的去中心化联邦学习策略，并使用五个联邦学习任务进行测试，分别是CNN模型训练fashion-mnist数据集、alexnet模型训练cifar10数据集、TextRnn模型训练THUsnews数据集、Resnet18模型训练SVHN数据集和LSTM模型训练sentiment140数据集。实验结果表明，设计的策略在五个任务非独立同分布的数据下进行去中心化联邦学习，准确率分别提升了2.51、5.16、17.58、2.46和5.23个百分点。

关键词: 区块链, 联邦学习, 深度学习, 数据隐私

TAN Rongjie, HONG Zhiyong, YU Wenhua, ZENG Zhiqiang. Decentralized Federated Learning Strategy for Non-Independent and Identically Distributed Data[J]. Computer Engineering and Applications, 2023, 59(1): 269-277.

谭荣杰, 洪智勇, 余文华, 曾志强. 非独立同分布数据下的去中心化联邦学习策略[J]. 计算机工程与应用, 2023, 59(1): 269-277.

References

[1] BOURSE F，MINELLI M，MINIHOLD M，et al.Fast homomorphic evaluation of deep discretized neural networks[C]//Advances in Cryptology-CRYPTO 2018，2018.
[2] SHI E，CHAN T H H，RIEFFEL E，et al.Distributed private data analysis[J].ACM Transactions on Algorithms，2017，13（4）：1-38.
[3] KIM H，PARK J，BENNIS M，et al.Blockchained on-device federated learning[J].IEEE Communications Letters，2020，24（6）：1279-1283.
[4] RAMANAN P，NAKAYAMA K.BAFFLE：Blockchain based aggregator free federated learning[C]//Proceedings of IEEE International Conference on Blockchain，2020：72-81.
[5] ZHOU S C，HUANG H W，CHEN W H，et al.PIRATE：A blockchain-based secure framework of distributed machine learning in 5G networks[J].IEEE Network，2020，34（6）：84-91.
[6] LI Y Z，CHEN C，LIU N，et al.A blockchain-based decentralized federated learning framework with committee consensus[J].IEEE Network，2021，35（1）：234-241.
[7] CAO M，ZHANG L，CAO B.Towards on-device federated learning：A direct acyclic graph-based blockchain approach[J].arXiv：2104.13092，2021.
[8] ZHAO Y，LI M，LAI L Z，et al.Federated learning with non-IID data [J].arXiv：1806.00582，2018.
[9] YOSHIDA N，NISHIO T，MORIKURA M，et al.Hybrid-Fl：Cooperative learning mechanism using non-iid data in wireless networks[J].arXiv：1905.07210，2019.
[10] WANG L P，WANG W，LI B.CMFL：Mitigating communication overhead for federated learning[C]//Proceedings of IEEE 39th International Conference on Distributed Computing Systems，2019：954-964.
[11] ONGARO D，OUSTERHOUT J.In search of an understandable consensus algorithm[C]//Proceedings of 2014 USENIX Annual Technical Conference，2014：305-320.
[12] KRIZHEVSKY A.Learning multiple layers of features from tiny images[J].Toronto，California：University of Toronto，2009：54-60.
[13] XIAO H，RASULK V.Fashion-MNIST：A novel image dataset for benchmarking machine learning algorithms[J].arXiv：1708.07747，2017.
[14] KRIZHEVSKY A，SUTSKEVER I，HINTON G E.ImageNet classification with deep convolutional neural networks[J].Communications of the ACM，2017，60（6）：84-90.
[15] LIU P，QIU X，HUANG X.Recurrent neural network for text classification with multi-task learning[C]//Proceedings of the 25th International Joint Conference on Artificial Intelligence，2016：2873-2879.
[16] SUN M S，LI J Y，GUO Z P，et al.THUCTC：An efficient Chinese text classification toolkit[DB/OL].（2016）[2022-03-15].http：//thuctc.thunlp.org/.
[17] HE K M，ZHANG X Y，REN S Q，et al.Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2016：770-778.
[18] NETZER Y，WANG T，COATES A，et al.Reading digits in natural images with unsupervised feature learning[C]//Proceedings of the NIPS Workshop on Deep Learning & Unsupervised Feature Learning，2011：12-47.
[19] HOCHREITER S，SCHMIDHUBER J.Long short-term memory[J].Neural Computation，1997，9（8）：1735-1780.
[20] GO A，BHAYANI R，LEI H.Twitter sentiment classification using distant supervision：cs224n Project Report[R].2009.