Methods and Prospects of Personalized Federated Learning

doi:10.3778/j.issn.1002-8331.2403-0207

Abstract

Abstract: Currently, with the advancement of artificial intelligence research, artificial intelligence is being widely adopted, and the increasing demand in areas such as data governance has led to growing awareness and concern for privacy protection, this has promoted the popularity of the federated learning (FL) framework. However, existing FL frameworks struggle to address heterogeneous issues and personalized user needs. In response to these challenges, methods of personalized federated learning (PFL) are studied and prospects are proposed. Firstly, the FL framework is outlined and its limitations are identified, leading to the research motivation for PFL based on FL scenarios. Subsequently, the analysis of statistical heterogeneity, model heterogeneity, communication heterogeneity, and device heterogeneity in PFL is conducted, and feasible solutions are proposed. Then, personalized algorithms in PFL such as client selection and knowledge distillation are categorized, and their innovations and shortcomings are analyzed. Finally, future research directions for PFL are discussed.

Key words: personalized federated learning (PFL), data governance, heterogeneity problems, privacy protection

摘要： 目前，随着人工智能研究的进步，人工智能被大规模采用，数据监管等领域的需求也促使人们对隐私保护的认识和关注越来越多，这促进了联邦学习（federated learning，FL）框架的流行。但现有的FL难以应对异构问题以及用户的个性化需求。为了应对上述问题，研究了个性化联邦学习（personalized federated learning，PFL）的相关方法并提出了展望。列举了FL的框架并指出了FL的不足，在FL场景的基础上，引出PFL的研究动机对PFL中的统计异构、模型异构、通信异构、设备异构进行分析并提出可行性方案；将PFL中的客户端选择、知识蒸馏等个性化算法分类并分析各自的创新与不足。最后，对PFL的未来研究方向进行了展望。

关键词: 个性化联邦学习（PFL）, 数据监管, 异构问题, 隐私保护

SUN Yanhua, WANG Zihang, LIU Chang, YANG Ruizhe, LI Meng, WANG Zhuwei. Methods and Prospects of Personalized Federated Learning[J]. Computer Engineering and Applications, 2024, 60(20): 68-83.

孙艳华, 王子航, 刘畅, 杨睿哲, 李萌, 王朱伟. 个性化联邦学习的相关方法与展望[J]. 计算机工程与应用, 2024, 60(20): 68-83.

References

[1] KAISSIS G A, MAKOWSKI M R, RüCKERT D, et al. Secure, privacy-preserving and federated machine learning in medical imaging[J]. Nature Machine Intelligence, 2020, 2(6): 305-311.
[2] VOIGT P, BUSSCHE V D A. The EU general data protection regulation (GDPR)[M]. Cham: Springer, 2017.
[3] YANG Q, LIU Y, CHEN T, et al. Federated machine learning: concept and applications[J]. ACM Transaction on Intelligent Systems and Technology, 2019, 10(2): 1-19.
[4] KAIROUZ P, MCMAHAN H B, AVENT B, et al. Advances and open problems in federated learning[J]. arXiv:1912.04977, 2019.
[5] FANG X, YE M. Robust federated learning with noisy and heterogeneous clients[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 10072-10081.
[6] HUANG W, YE M, DU B. Learn from others and be yourself in heterogeneous federated learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 10143-10153.
[7] KAISSIS G, ZILLER A, PASSERAT-PALMBACH J, et al. End-to-end privacy preserving deep learning on multi-institutional medical imaging[J]. Nature Machine Intelligence, 2021, 3(6): 473-484.
[8] USYNIN D, ZILLER A, MAKOWSKI M, et al. Adversarial interference and its mitigations in privacy-preserving collaborative machine learning[J]. Nature Machine Intelligence, 2021, 3(9): 749-758.
[9] MCMAHAN B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[C]//Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 2017: 1273-1282.
[10] DRAINAKIS G, KATSAROS K V, PANTAZOPOULOS P, et al. Federated vs centralized machine learning under privacy-elastic users: a comparative analysis[C]//Proceedings of the 2020 IEEE 19th International Symposium on Network Computing and Applications, 2020: 1-8.
[11] WU Q, HE K, CHEN X. Personalized federated learning for intelligent IoT applications: a cloud-edge based framework[J]. IEEE Open Journal of the Computer Society, 2020, 1: 35-44.
[12] KARIMIREDDY S P, KALE S, MOHRI M, et al. Scaffold: stochastic controlled averaging for federated learning[C]//Proceedings of the International Conference on Machine Learning, 2020: 5132-5143.
[13] SIM K C, ZADRAZIL P, BEAUFAYS F. An investigation into on-device personalization of end-to-end automatic speech recognition models [J]. arXiv:1909.06678, 2019.
[14] SUN Y, OCHIAI H. Homogeneous learning: self-attention decentralized deep learning[J]. IEEE Access, 2022, 10: 7695-7703.
[15] HOSSEINALIPOUR S, BRINTON C G, AGGARWAL V, et al. From federated to fog learning: distributed machine learning over heterogeneous wireless networks[J]. IEEE Communications Magazine, 2020, 58(12): 41-47.
[16] ZHOU Y, LI Z, TANG T, et al. Depersonalized federated learning: gradient descent[C]//Proceedings of the 2023 IEEE International Conference on Communications, 2023: 1988-1993.
[17] HSIEH K, PHANISHAYEE A, MUTLU O, et al. The Non-IID data quagmire of decentralized machine learning[C]//Proceedings of the International Conference on Machine Learning, 2020: 4387-4398.
[18] LI T, SAHU A K, ZAHEER M, et al. Federated optimization in heterogeneous networks[J]. arXiv:1812.06127, 2018.
[19] WANG X, CHEN W, XIA J, et al. HetVis: a visual analysis approach for identifying data heterogeneity in horizontal federated learning[J]. IEEE Transactions on Visualization and Computer Graphics, 2022, 29(1): 310-319.
[20] ZHOU X, LEI X, YANG C, et al. Handling data heterogeneity for IoT devices in federated learning: a knowledge fusion approach[J]. IEEE Internet of Things Journal, 2024, 11(5): 8090-8104.
[21] ZHANG J, LI Z, LI B, et al. Federated learning with label distribution skew via logits calibration[C]//Proceedings of the International Conference on Machine Learning, 2022: 26311-26329.
[22] KUWIL F H. A new feature extraction approach of medical image based on data distribution skew[J]. Neuroscience Informatics, 2022, 2(3): 100097.
[23] JIA M, WU Z, REITER A, et al. Intentonomy: a dataset and study towards human intent understanding[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 12986-12996.
[24] YANG S, PARK H, BYUN J, et al. Robust federated learning with noisy labels[J]. IEEE Intelligent Systems, 2022, 37(2): 35-43.
[25] LUO Z, WANG Y, WANG Z, et al. Disentangled federated learning for tackling attributes skew via invariant aggregation and diversity transferring[J]. arXiv:2206.06818, 2022.
[26] SARANYA A, ABINESH S, CHIDAMBARAM S. A performance analysis of predicting diabetics disease using hybrid machine learning approach[C]//Proceedings of the 2023 9th International Conference on Advanced Computing and Communication Systems, 2023: 1105-1108.
[27] ZHU H, XU J, LIU S, et al. Federated learning on Non-IID data: a survey[J]. Neurocomputing, 2021, 465: 371-390.
[28] SHANG X, LU Y, HUANG G, et al. Federated learning on heterogeneous and long-tailed data via classifier re-training with federated features[J]. arXiv:2204.13399, 2022.
[29] HUANG W, YE M, DU B, et al. Few-shot model agnostic federated learning[C]//Proceedings of the 30th ACM International Conference on Multimedia, 2022: 7309-7316.
[30] LI Q, DIAO Y, CHEN Q, et al. Federated learning on Non-IID data silos: an experimental study[C]//Proceedings of the 2022 IEEE 38th International Conference on Data Engineering, 2022: 965-978.
[31] YE M, FANG X, DU B, et al. Heterogeneous federated learning: state-of-the-art and research challenges[J]. ACM Computing Surveys, 2023, 56(3): 1-44.
[32] DENG Y, CHEN W, REN J, et al. TailorFL: dual-personalized federated learning under system and data heterogeneity[C]//Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems, 2022: 592-606.
[33] CHO Y J, WANG J, CHIRVOLU T, et al. Communication-efficient and model-heterogeneous personalized federated learning via clustered knowledge transfer[J]. IEEE Journal of Selected Topics in Signal Processing, 2023, 17(1): 234-247.
[34] DIAO E, DING J, TAROKH V. HeteroFL: computation and communication efficient federated learning for heterogeneous clients[J]. arXiv:2010.01264, 2020.
[35] JANG J, HA H, JUNG D, et al. FedClassAvg: local representation learning for personalized federated learning on heterogeneous neural networks[C]//Proceedings of the 51st International Conference on Parallel Processing, 2022: 1-10.
[36] ZHANG J, GUO S, GUO J, et al. Towards data-independent knowledge transfer in model-heterogeneous federated learning[J]. IEEE Transactions on Computers, 2023, 72(10): 2888-2901.
[37] SHEN L, ZHENG Y. FedDM: data and model heterogeneity-aware federated learning via dynamic weight sharing[C]//Proceedings of the 2023 IEEE 43rd International Conference on Distributed Computing Systems, 2023: 975-976.
[38] CHEN D, YAO L, GAO D, et al. Efficient personalized federated learning via sparse model-adaptation[C]//Proceedings of the International Conference on Machine Learning, 2023: 5234-5256.
[39] DUAN J, DUAN J, WAN X, et al. Efficient federated learning method for cloud-edge network communication[C]//Proceedings of the 2023 5th International Conference on Communications, Information System and Computer Engineering, 2023: 118-121.
[40] WANG J, LI Y, LIANG M, et al. Embedding representation of academic heterogeneous information networks based on federated learning[C]//Proceedings of the 2022 IEEE 8th International Conference on Cloud Computing and Intelligent Systems , 2022: 516-520.
[41] KHAN A N, RIZWAN A, AHMAD R, et al. Hetero-FedIoT: a rule-based interworking architecture for heterogeneous federated IoT networks[J]. IEEE Internet of Things Journal, 2024, 11(4): 5920-5938.
[42] GAO D, WANG H, GUO X Z, et al. Federated learning based on CTC for heterogeneous internet of things[J]. IEEE Internet of Things Journal, 2023, 10(24): 22673-22685.
[43] YANG C, XU M, WANG Q, et al. FLASH: heterogeneity-aware federated learning at scale[J]. IEEE Transactions on Mobile Computing, 2024, 23(1): 483-500.
[44] SUN Y, SHAO J, MAO Y, et al. Semi-decentralized federated edge learning with data and device heterogeneity[J]. IEEE Transactions on Network and Service Management, 2023, 20(2): 1487-1501.
[45] PILLAY S, MACDONALD A J, BRITO R, et al. Federated learning on edge devices in a lunar analogue environment[C]//Proceedings of the 2023 IEEE International Geoscience and Remote Sensing Symposium, 2023: 2006-2009.
[46] BAUGHMAN M, HUDSON N, FOSTER I, et al. Balancing federated learning trade-offs for heterogeneous environments[C]//Proceedings of the 2023 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events, 2023: 404-407.
[47] ZHAO L, HU S, SHI Z. Federated learning scheme based on gradient compression and local differential privacy[C]//Proceedings of the 2023 IEEE International Conference on Control, Electronics and Computer Technology, 2023: 1567-1572.
[48] IQBAL M, TARIQ A, ADNAN M, et al. FL-ODP: an optimized differential privacy enabled privacy preserving federated learning[J]. IEEE Access, 2023, 11: 116674-116683.
[49] HU J, WANG Z, SHEN Y, et al. Shield against gradient leakage attacks: adaptive privacy-preserving federated learning[J]. IEEE/ACM Transactions on Networking, 2024, 32(2): 1407-1422.
[50] QAYYUM T, CHEN S. Decentralized two-stage federated learning with knowledge transfer[C]//Proceedings of the 2023 IEEE International Conference on Communications, 2023: 3181-3186.
[51] HUANG W, YE M, SHI Z, et al. Generalizable heterogeneous federated cross-correlation and instance similarity learning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, 42(2): 712-728.
[52] LI L, DUAN M, LIU D, et al. FedSAE: a novel self-adaptive federated learning framework in heterogeneous systems[C]//Proceedings of the 2021 International Joint Conference on Neural Networks, 2021: 1-10.
[53] QIN Z, YANG L, WANG Q, et al. Reliable and interpretable personalized federated learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023: 20422-20431.
[54] TANG M, NING X, WANG Y, et al. FedCor: correlation-based active client selection strategy for heterogeneous federated learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 10102-10111.
[55] LIM H, TANJUNG S Y, IWAN I, et al. Fed-Sic: selecting important clients for federated learning[C]//Proceedings of the 31st Signal Processing and Communications Applications Conference, 2023: 1-4.
[56] MAO W, LU X, JIANG Y, et al. Joint client selection and bandwidth allocation of wireless federated learning by deep reinforcement learning[J]. IEEE Transactions on Services Computing, 2024, 17(1): 336-348.
[57] FINN C, ABBEEL P, LEVINE S. Model-agnostic meta-learning for fast adaptation of deep networks[C]//Proceedings of the International Conference on Machine Learning, 2017: 1126-1135.
[58] CHU Y W, HOSSEINALIPOUR S, TENORIO E, et al. Multi-layer personalized federated learning for mitigating biases in student predictive analytics[J]. arXiv:2212.02985, 2022.
[59] YU F, LIN H, WANG X, et al. Communication-efficient personalized federated meta-learning in edge networks[J]. IEEE Transactions on Network and Service Management, 2023, 22(2): 1558-1571.
[60] DINH T C, TRAN N, NGUYEN J. Personalized federated learning with moreau envelopes[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems, 2020: 21394-21405.
[61] LI D, WANG J. FedMD: heterogenous federated learning via model distillation[J]. arXiv:1901.03581, 2019.
[62] LI C, LI G, VARSHNEY P K. Decentralized federated learning via mutual knowledge transfer[J]. IEEE Internet of Things Journal, 2021, 9(2): 1136-1147.
[63] SHEN J, YANG S, ZHAO C, et al. FedLED: label-free equipment fault diagnosis with vertical federated transfer learning[J]. IEEE Transactions on Instrumentation and Measurement, 2024, 73: 1-10.
[64] DANG Q, ZHANG G, WANG L, et al. Hybrid IoT device selection with knowledge transfer for federated learning[J]. IEEE Internet of Things Journal, 2024, 11(7): 12216-12227.
[65] SMITH V, CHIANG C K, SANJABI M, et al. Federated multi-task learning[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017: 4427-4437.
[66] LI T, HU S, BEIRAMI, et al. Ditto: fair and robust federated learning through personalization[C]//Proceedings of the International Conference on Machine Learning, 2021: 6357-6368.
[67] AI Y, CHEN Q, LIANG Y, et al. Communication-efficient federated multi-task learning with sparse sharing[C]//Proceedings of the IEEE 34th Annual International Symposium on Personal, Indoor and Mobile Radio Communications, 2023: 1-6.
[68] SINGH A, CHANDRASEKAR S, SEN T, et al. Federated multi-task learning for complaint identification using graph attention network[J]. IEEE Transactions on Artificial Intelligence, 2024, 5(3): 1277-1286.
[69] YE M, SHEN J, ZHANG X, et al. Augmentation invariant and instance spreading feature for softmax embedding[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(2): 924-939.
[70] LI Q, HE B, SONG D. Model-contrastive federated learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021: 10713-10722.
[71] ZHUANG W, GAN X, WEN Y, et al. Collaborative unsupervised visual representation learning from decentralized data[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021: 4912-4921.
[72] ZHAO C, GAO Z, YANG Y, et al. FedUSC: collaborative unsupervised representation learning from decentralized data for internet of things[J]. IEEE Internet of Things Journal, 2023, 10(15): 13601-13611.
[73] JIN X, BU J, YU Z, et al. FedCrack: federated transfer learning with unsupervised representation for crack detection[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(10): 11171-11184.
[74] ZHANG L, SHEN L, DING L, et al. Fine-tuning global model via data-free knowledge distillation for Non-IID federated learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 10174-10183.
[75] XU Y, FAN H. FedDK: improving cyclic knowledge distillation for personalized healthcare federated learning [J]. IEEE Access, 2023, 11: 72409-72417.
[76] QI P, ZHOU X, DING Y, et al. FedBKD: heterogenous federated learning via bidirectional knowledge distillation for modulation classification in IoT-edge system[J]. IEEE Journal of Selected Topics in Signal Processing, 2022, 17(1): 189-204.
[77] GOU J, SUN L, YU B, et al. Collaborative knowledge distillation via multiknowledge transfer[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(5): 6718-6730.
[78] SATTLER F, MüLLER K R, SAMEK W. Clustered federated learning: model-agnostic distributed multitask optimization under privacy constraints[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 32(8): 3710-3722.
[79] DUAN M, LIU D, JI X, et al. FedGroup: efficient federated learning via decomposed similarity-based clustering[C]//Proceedings of the 2021 IEEE International Conference on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking, 2021: 228-237.
[80] LIM W Y B, NG J S, XIONG Z, et al. Dynamic edge association and resource allocation in self-organizing hierarchical federated learning networks[J]. IEEE Journal on Selected Areas in Communications, 2021, 39(12): 3640-3653.
[81] MEHTA M, SHAO C. A greedy agglomerative framework for clustered federated learning[J]. IEEE Transactions on Industrial Informatics, 2023, 19(12): 11856-11867.
[82] XU B, XA W, ZHAO H, et al. Clustered federated learning in internet of things: convergence analysis and resource optimization[J]. IEEE Internet of Things Journal, 2023, 11(2): 3217-3232.
[83] HUANG X, LIU J, LAI Y, et al. EEFED: personalized federated learning of execution & evaluation dual network for CPS intrusion detection[J]. IEEE Transactions on Information Forensics and Security, 2023, 18: 41-56.
[84] FIRDAUS M, RHEE K H. Towards trustworthy collaborative healthcare data sharing[C]//Proceedings of the 2023 IEEE International Conference on Bioinformatics and Biomedicine, 2023: 4059-4064.
[85] LIAN Z, WANG W, HAN Z, et al. Blockchain-based personalized federated learning for internet of medical things[J]. IEEE Transactions on Sustainable Computing, 2023, 8(4): 694-702.
[86] ZHENG X, NAGHIZADEH P, YENER A. DiPLe: learning directed collaboration graphs for peer-to-peer personalized learning[C]//Proceedings of the 2022 IEEE Information Theory Workshop, 2022: 446-451.
[87] ZHAN Y, ZHANG J, HONG Z, et al. A survey of incentive mechanism design for federated learning[J]. IEEE Transactions on Emerging Topics in Computing, 2021, 10(2): 1035-1044.
[88] WANG Z, KUANG W, XIE Y, et al. FederatedScope-GNN: towards a unified, comprehensive and efficient package for federated graph learning[C]//Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2022: 4110-4120.
[89] MATSUDA K, SASAKI Y, XIAO C, et al. Benchmark for personalized federated learning[J]. IEEE Open Journal of the Computer Society, 2024, 5: 2-13.