联邦学习中的攻击手段与防御机制研究综述

doi:10.3778/j.issn.1002-8331.2306-0243

摘要/Abstract

摘要： 联邦学习的攻防技术是联邦学习系统安全的核心问题。联邦学习的攻防技术能大幅降低联邦学习系统被攻击的风险，明显提升联邦学习系统的安全性。深入了解联邦学习的攻防技术，可以推进联邦学习领域的研究，实现联邦学习的广泛应用。因此，对联邦学习的攻防技术进行研究具有十分重要的意义。简要地介绍了联邦学习的概念、基本工作流程、类型及可能存在的安全问题；介绍联邦学习系统可能遭受到的攻击，梳理了相关研究；从联邦学习系统有无目标性的防御措施出发，将防御措施分为通用性防御措施及针对性防御措施两类，并对其进行了针对性的总结；对联邦学习安全性未来的研究方向进行了梳理与分析，为相关研究者在联邦学习安全性方面的研究工作提供了参考。

关键词: 联邦学习, 攻击手段, 防御措施, 隐私保护

Abstract: The attack and defense techniques of federated learning are the core issue of federated learning system security. The attack and defense techniques of federated learning can significantly reduce the risk of being attacked and greatly enhance the security of federated learning systems. Deeply understanding the attack and defense techniques of federated learning can advance research in the field and achieve its widespread application of federated learning. Therefore, it is of great significance to study the attack and defense techniques of federated learning. Firstly, this paper briefly introduces the concept, basic workflow, types, and potential existing security issues of federated learning. Subsequently, the paper introduces the attacks that the federated learning system may encounter, and relevant research is summarized during the introduction. Then, starting from whether the federated learning system has targeted defense measures, the defense measures are divided into two categories：universal defense measures and targeted defense measures, and targeted summary are made. Finally, it reviews and analyzes the future research directions for the security of federated learning, providing reference for relevant researchers in their research work on the security of federated learning.

Key words: federated learning, attack method, defense mechanism, privacy protection

张世文, 陈双, 梁伟, 李仁发. 联邦学习中的攻击手段与防御机制研究综述[J]. 计算机工程与应用, 2024, 60(5): 1-16.

ZHANG Shiwen, CHEN Shuang, LIANG Wei, LI Renfa. Survey on Attack Methods and Defense Mechanisms in Federated Learning[J]. Computer Engineering and Applications, 2024, 60(5): 1-16.

参考文献

[1] 张思思, 高旭光, 滑文强. 基于聚类与人工神经网络的遥感图像信息提取方法[J]. 电子设计工程, 2020, 28(15): 106-109.
ZHANG S S, GAO X G, HUA W Q. Remote sensing image information extraction method based on clustering and artificial neural network[J]. International Electronic Elements, 2020, 28(15): 106-109.
[2] MCMAHAN H B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data[C]//Proceeding of the 20th International Conference on Artificial Intelligence and Statistics, Ft Lauderdale FL, April 20-22, 2017. USA: JMLR, 2017: 1273-1282.
[3] JAGIELSKI M, OPREA A, BIGGIO B, et al. Manipulating machine learning: poisoning attacks and countermeasures for regression learning[C]//Proceeding of the 39th IEEE Symposium on Security and Privacy, San Francisco, May 21-23, 2018. NJ: IEEE, 2018: 19-35.
[4] WANG Z B, SONG M K, ZHANG Z F, et al. Beyond inferring class representatives: user-level privacy leakage from federated learning[C]//Proceeding of the 38th Annual IEEE International Conference on Computer Communications, Paris, April 29-May 2, 2019. NJ: IEEE, 2019: 2512-2520.
[5] HITAJ B, ATENIESE G, PEREZ-CRUZ F. Deep models under the GAN: information leakage from collaborative deep learning[C]//Proceeding of the 2017 ACM SIGSAC Conference on Computer and Communications Security, Dallas, TX, October, 2017. New York: ACM, 2017: 603-618.
[6] YANG Q, LIU Y, CHEN T J, et al. Federated machine learning[J]. ACM Transactions on Intelligent Systems and Technology, 2019, 10(2): 1-19.
[7] LI T, SAHU A K, TALWALKARA, et al. Federated learning: challenges, methods, and future directions[J]. IEEE Signal Processing Magazine, 2020, 37(3): 50-60.
[8] KAIROZ P, MCMAHAN H B, AVENT B, at al. Advances and open problems in federated learning[J]. Foundations and Trends? in Machine Learning, 2021, 14(1/2): 1-210.
[9] MCMAHAN H B, MOORE E, RAMAGE D, et al. Federated learning of deep networks using model averaging[J]. arXiv:1602.05629,2016.
[10] 刘艺璇, 陈红, 刘宇涵, 等. 联邦学习中的隐私保护技术[J]. 软件学报, 2022, 33(3): 1057-1092.
LIU Y X, CHEN H, LIU Y H, et al. Privacy-preserving techniques in federated learning[J]. Journal of Software, 2022, 33(3): 1057-1092.
[11] PAN S J, YANG Q. A Survey on transfer learning[J]. IEEE Transactions on Knowledge and Data Engineering, 2010, 22(10): 1345-1359.
[12] 何英哲, 胡兴波, 何锦雯, 等. 机器学习系统的隐私和安全问题综述[J]. 计算机研究与发展, 2019, 56(10): 2049-2070.
HE Y Z, HU X B, HE J W, et al. Privacy and security issues in machine learning systems: a survey[J]. Journal of Computer Research and Development, 2019, 56(10): 2049-2070.
[13] BIGGIO B, NELSON B, LASKOV P. Poisoning attacks against support vector machines[J]. arXiv:1206.6389,2012.
[14] RUBINSTEIN B, NELSON B, LING H, et al. ANTIDOTE: understanding and defending against poisoning of anomaly detectors[C]//Proceeding of the 9th ACM SIGCOMM Conference on Internet measurement, Chicago, Nov 4-6, 2009. New York: ACM, 2009: 1-14.
[15] MUOZ-GONZALEZ L, BIGGIO B, DEMONTIS A, et al. Towards poisoning of deep learning algorithms with back-gradient optimization[J]. ACM, 2017, 17: 27-38.
[16] SUN G, CONG Y, DONG J, et al. Data poisoning attacks on federated machine learning[J]. IEEE Internet of Things Journal, 2022, 9(13): 11365-11375.
[17] TOLPEGIN V, TRUEX S, GURSOY M E, et al. Data poisoning attacks against federated learning systems[C]//Proceeding of the ESORICS 2020, UK, September 14-18, 2020. Berlin: Springer, 2020: 480-501.
[18] LI Z, WU X K, JIANG C J. Efficient poisoning attacks and defenses for unlabeled data in DDoS prediction of intelligent transportation systems[J]. Security and Safety, 2022, 1: 145-165.
[19] ZHOU X C, XU M, WU Y M, et al. Deep model poisoning attack on federated learning[J]. Future Internet, 2021, 13(3): 73.
[20] HOSSAIN M T, ISLAM S, BADSHA B et al. DeSMP: differential privacy-exploited stealthy model poisoning attacks in federated learning[C]//Proceeding of the 17th International Conference on Mobility, Sensing and Networking (MSN), Exeter, Dec 13-15, 2021. NJ: IEEE, 2021: 167-174.
[21] CAO X, GONG N Z. MPAF: model poisoning attacks to federated learning based on fake clients[C]//Proceeding of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, LA, USA, June 19-20, 2022. NJ: IEEE, 2022.
[22] ZHAO S, MA X, ZHENG X, et al. Clean-label backdoor attacks on video recognition models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, June 13-19, 2020. NJ: IEEE, 2020: 14431-14440.
[23] BHAGOJI A N, CHAKRABORTY S, MITTAL P, et al. Analyzing federated learning through an adversarial lens[C]//Proceedings of the 36th International Conference on Machine Learning, Long Beach, June 9-15, 2019. Germany: Statistics, 2019: 1467-5463.
[24] SZEGEDY C, ZAREMBA W, SUTSKEVER I, et al. Intriguing properties of neural networks[C]//Proceeding of the 2nd International Conference on Learning Representations, Banff, Canada, Apr 14-16, 2014.
[25] 张思思, 左信, 刘建伟. 深度学习中的对抗样本问题[J]. 计算机学报, 2019, 42(8): 1886-1904.
ZHANG S S, ZUO X, LIU J W. the problem of the adversarial examples in deep learning[J]. Chinese Journal of Computers, 2019, 42(8): 1886-1904.
[26] LING X, JI L, ZOU J, et al. DEEPSEC: a uniform platform for security analysis of deep learning model[C]//Proceeding of the 2019 IEEE Symposium on Security and Privacy (SP), San Francisco, CA, USA, May 19-23, 2019. NJ: IEEE, 2019: 673-690.
[27] PAPERNOT N, MCDANIEL P, GOODFELLOW I. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples[J]. arXiv:1605.07277,2016.
[28] ZHANG Y H, JIA R X, PEI H Z, et al. The secret revealer: generative model inversion attacks against deep neural networks[C]//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Seattle, WA, USA, June 13-19, 2020. NJ: IEEE, 2020: 250-258.
[29] REN H C, DENG J J, XIE X H. GRNN: generative regression neural network—a data leakage attack for federated learning[J]. ACM Transactions on Intelligent Systems and Technology, 2022, 13(4): 1-24.
[30] 孔锐, 蔡佳纯, 黄钢. 基于生成对抗网络的对抗攻击防御模型[J/OL]. 自动化学报(2020-07-23). http://www.aas.net.cn/cn/article/doi/10.16383/j.aas.2020.c200033?viewType=HTML.
KONG R, CAI J C, HUANG G. Defense to adversarial attack with generative adversarial network[J/OL]. Acta Automatica Sinica (2020-07-23). http://www.aas.net.cn/cn/article/doi/10.16383/j.aas.2020.c200033?viewType=HTML.
[31] BARRENO M, NELSON B, SEARS R, et al. CAN machine learning be secure[C]//Proceedings of the 2006 ACM Symposium on Information, Computer and Communications Security, Taipei, China, March 21-24, 2006. New York: ACM Press, 2006.
[32] LEE H, KIM J, HUSSA IN R, et al. On defensive neural networks against inference attack in federated learning[C]//Proceedings of 2021 IEEE International Conference on Communications, Seoul, Korea, June 14-23, 2021. NJ: IEEE, 2021: 1-6.
[33] SHOKRI R, STRONATI M, SONG C, et al. Membership inference attacks against machine learning models[C]//Proceedings of IEEE Symposium on Security and Privacy (SP), San Jose, May 22-26, 2017. NJ: IEEE, 2017: 3-18.
[34] AONO Y, HAYASHI T, PHONG L T, et al. Scalable and secure logistic regression via homomorphic encryption[C]//Proceedings of the 6th ACM Conference on Data and Application Security and Privacy, New Orleans Louisiana USA, March 9-11. New York: ACM, 2016: 142-144.
[35] MELIS L, SONG C Z, CRISTOFARO E D, et al. Exploiting unintended feature leakage in collaborative learning[C]//Proceeding of the IEEE Symposium on Security and Privacy, San Francisco, May 19-23, 2019. NJ: IEEE, 2019: 691-706.
[36] NASR M, SHOKRI R, HOUMANSADR A. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning[C]//Proceeding of the IEEE Symposium on Security and Privacy, San Francisco, May 19-23, 2019. NJ: IEEE, 2019.
[37] CHEN J L, ZHANG J L, ZHAO Y C, et al. Beyond model-level membership privacy leakage: an adversarial approach in federated learning[C]//Proceedings of the 29th International Conference on Computer Communications and Networks, Honolulu, Aug 3-6, 2020. NJ: IEEE, 2020: 1-9.
[38] ZHANG J W, ZHANG J L, CHEN J J, et al. GAN enhanced membership inference: A passive local attack in federated learning[C]//Proceedings of the IEEE International Conference on Communications, Dublin, June 7-11, 2020. NJ: IEEE, 2020: 1-6.
[39] SONG M K, WANG Z B, ZHANG Z F, et al. Analyzing user-level privacy attack against federated learning[J]. IEEE Journal on Selected Areas in Communications, 2020, 38(10): 2430-2444.
[40] ZHU L G, LIU Z J, HAN S. Deep leakage from gradients[C]//Proceeding of the 33rd International Conference on Neural Information Processing Systems, NY, December 8-14, 2019. Berlin: Springer, 2019: 14747-14756.
[41] SHEN M, WANG H, ZHANG B, et al. Exploiting unintended property leakage in blockchain-assisted federated learning for intelligent edge computing[J]. IEEE Internet of Things Journal, 2021, 8(4): 2265-2275.
[42] TRAMER F, ZHANG F, JUELS A, et al. Stealing machine learning models via prediction APIs[C]//Proceedings of the 25th USENIX Conference on Security Symposium, Austin, May 31, 2019. USA: USENIX Association, 2016: 601-618.
[43] WANG B, GONG N Z. Stealing hyperparameters in machine learning[C]//Proceedings of 2018 IEEE Symposium on Security and Privacy (SP), San Francisco, May 20-24, 2018. NJ: IEEE, 2018: 36-52.
[44] OH S J, AUGUSTIN M, SCHIELE B, et al. Towards reverse engineering black-box neural networks[J]. arXiv:1711.01768,2017.
[45] FREDRIKSON M, JHA S, RISTENPART T. Model inversion attacks that exploit confidence information and basic counter measures[C]//Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Denver, Colorado, October 12-16, 2015. New York: Association for Computing Machinery, 2015: 1322-1333.
[46] ATENIESE G, MANCINI L V, SPOGNARDI A, et al. Hacking smart machines with smarter ones: how to extract meaningful data from machine learning classifiers[J]. International Journal of Security and Networks, 2015, 10(3): 137-150.
[47] LYU L J, YU H, YANG Q. Threats to federated learning: a survey[J]. arXiv:2003.02133,2020.
[48] BOUACIDA N, MOHAPATRA P. Vulnerabilities in federated learning[J]. IEEE Access, 2021, 9: 63229-63249.
[49] MOTHUKURI V, PARIZIE M, POURIYEH S, et al. A survey on security and privacy of federated learning[J]. Future Generation Computer Systems, 2021, 115: 619-640.
[50] DWORK C, MCSHERRY F, NISSIM K, et al. Calibrating noise to sensitivity in private data analysis[C]//Lecture Notes in Computer Science, NY, March 4-7, 2006. Berlin: Springer, 2006: 265-284.
[51] BASSILY R, NISSIM K, STEMMER U, et al. Practical locally private heavy hitters[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, December 4-9, 2017. Red Hook, NY, USA: Curran Associates Inc, 2017: 2285-2293.
[52] BITTAU A, ERLINGSSON L, MANIATIS P, et al. Prochlo: strong privacy for analytics in the crowd[J]. Journal of Machine Learning Research, 2020, 21(1): 1532-4435.
[53] ERLINGSSON L, FELDMAN V, MIRONOV I, et al. Amplification by shuffling: from local to central differential privacy via anonymity[C]//Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms, San Diego, Jan 6-9, 2019. USA: Society for Industrial and Applied Mathematics, 2019: 2468-2479.
[54] CHEU A, SMITH A, ULLMAN J, et al. Distributed differential privacy via shuffling[J]. arXiv:1808.01394,2018.
[55] AVENT B, KOROLOVA A, ZEBER D, et al. BLENDER: enabling local search with a hybrid differential privacy model[J]. Journal of Privacy and Confidentiality, 2017, 9(2): 2575-8527.
[56] MCMAHAN HB, RAMAGE D, TALWAR K. Learning differentially private recurrent language models[J]. arXiv:1710.
06963,2017.
[57] CHOUDHURY O, GKOULALAS-DIVANIS A, SALONIDIS T, et al. Differential privacy-enabled federated learning for sensitive health data[J]. arXiv:1910.02578,2019.
[58] GEYER R C, KLEIN T, NABI M. Differentially private federated learning: a client level perspective[J]. arXiv:1712.07557,2017.
[59] BHOWMICK A, DUCHI J, FREUDIGER J, et al. Protection against reconstruction and its applications in private federated learning[J]. arXiv:1812.00984,2018.
[60] ABADI M, CHU A, GOODFELLOW I, et al. Deep learning with differential privacy[C]//Proceedings of the 2016 ACM SIGASC Conference on Computer and Communications Security, NY, October 24-28, 2016, NY: ACM, 2016: 308-318.
[61] RIVEST R L, ADLEMAN L, DERTOUZOS M L. On data banks and privacy homomorphisms[J]. Foundations of Secure Computation, 1978, 4(11): 169-180.
[62] BONEH D, GOH E J, NISSIM K. Evaluating 2-DNF formulas on ciphertexts[C]//Proceedings of 24th Annual International Conference on the Theory and Applications of Cryptographic Techniques, Aarhus, Denmark, May 22-26, 2005. Berlin: Springer, 2005: 325-341.
[63] FAN J, VERCAUTEREN F. Somewhat practical fully homomorphic encryption[J]. IACR Cryptology Eprint Archive, 2012, 2012: 144.
[64] BRAKERSKI Z, GENTRY C, VAIKUNTANATHAN V. (Leveled) fully homomorphic encryption without bootstrapping[J]. ACM Transactions on Computation Theory (TOCT), 2014, 6(3): 1-36.
[65] CHEON J H, KIM A, KIM M, et al. Homomorphic encryption for arithmetic of approximate numbers[C]//Proceedings of 23rd International Conference on the Theory and Application of Cryptology and Information Security, Hong Kong, China, December 3-7, 2017. Berlin: Springer, 2017: 409-437.
[66] MADI A, STAN O, MAYOUE A, et al. A secure federated learning framework using homomorphic encryption and verifiable computing[C]//Proceedings of 2021 Reconciling Data Analytics, Automation, Privacy, and Security: A Big Data Challenge, Hamilton, Ontario, May 18-19, 2021. NJ: IEEE, 2020: 1-8.
[67] PHONG L T, AONO Y, HAYASHI T, et al. Privacy-preserving deep learning via additively homomorphic encryption[J]. IEEE Transactions on Information Forensics and Security, 2018, 13(5): 1333-1345.
[68] REYZIN L, SMITH A, YAKOUBOV S. Turning HATE into LOVE: compact homomorphic Ad Hoc threshold encryption for scalable MPC[C]//Proceedings of 5th International Symposium on Cyber Security Cryptography and Machine Learning, Be'er Sheva, July 8-9, 2021. Berlin: Springer, 2021: 361-378.
[69] ROTH E, NOBLE D, FALK BH, et al. Honeycrisp: largescale differentially private aggregation without a trusted core[C]//Proceedings of the 27th ACM Symposium on Operating Systems Principles, Huntsville, Ontario, October 27-30, 2019. NY: ACM, 2019: 196-210.
[70] SHAMIR A. How to share a secret[J]. ACM, 1979, 22(11): 612-613.
[71] BLAKLEY G R. Safeguarding cryptographic keys[C]//Proceedings of International Workshop on Managing Requirements Knowledge (MARK), New York, June 4-7, 1979. NJ: IEEE, 1979: 313-318.
[72] ASMUTH C, BLOOM J. A modular approach to key safeguarding[J]. IEEE Transactions on Information Theory, 1983, 29(2): 208-210.
[73] BONAWITZ K, IVANOV V, KREUTER B, et al. Practical secure aggregation for federated learning on user-held data[J]. arXiv:1611.04482,2016.
[74] HAN G, ZHANG T T, ZHANG Y H, et al. Verifiable and privacy preserving federated learning without fully trusted centers[J]. Journal of Ambient Intelligence and Humanized Computing, 2022, 13(3): 1431-1441.
[75] FEREIDOONI H, MARCHAL S, MIETTINEN M, et al. SAFELearn: secure aggregation for private federated learning[C]//Proceedings of 2021 IEEE Security and Privacy Workshops (SPW), San Francisco, May 27, 2021. NJ: IEEE, 2021: 56-62.
[76] BARACALDO N, CHEN B, LUDWIG H, et al. Mitigating poisoning attacks on machine learning models: a data provenance based approach[C]//Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, Dallas, Texas, November 3, 2017. NY: ACM, 2017: 103-110.
[77] ANDREINA S, MARSON G A, MLLERING H, et al. BaFFLe: backdoor detection via feedback-based federated learning[C]//Proceedings of IEEE 41st International Conference on Distributed Computing Systems (ICDCS), DC, July 7-10, 2021. NJ: IEEE, 2021: 52-863.
[78] MIYATA T, MAEDA S, KOYAMA M, et al. Virtual adversarial training: a regularization method for supervised and semi-supervised learning[J]. IEEE Transactions on Pattern Analysis And Machine Intelligence, 2018, 41(8): 1979-1993.
[79] NAIR A K, RAJ E D, SAHOO J. A robust analysis of adversarial attacks on federated learning environments[J]. Computer Standards & Interfaces, 2023, 86: 103723.
[80] LIANG B, LI H C, SU M Q, et al. Detecting adversarial image examples in deep neural networks with adaptive noise reduction[J]. IEEE Transactions on Dependable and Secure Computing, 2021, 18(1): 72-85.
[81] SHAH D, DUBE P, CHAKRABORTY S, et al. Adversarial training in communication constrained federated learning[J]. arXiv:2103.01319,2021.
[82] SO J, GULER B, AVESTIMEHR A S. Turbo-aggregate: breaking the quadratic aggregation barrier in secure federated learning[J]. IEEE Journal on Selected Areas in Information Theory, 2021, 2(1): 479-489.
[83] CHEN Y, LUO F, LI T, et al. A training-integrity privacy-preserving federated learning scheme with trusted execution environment[J]. Information Sciences, 2020, 522: 69-79.