Diagnosing Machine Learning Problems in Federated Learning Systems: A Case Study

• Autorzy: Bogacka Karolina , Danilenka Anastasiya , Wasielewska-Michniewska Katarzyna

Identyfikatory

DOI

10.15439/2023F722

• Języki publikacji: EN

Abstrakty

EN

The proliferation of digital artifacts with various computing capabilities, along with the emergence of edge computing, offers new possibilities for the development of Machine Learning solutions. These new possibilities have led to the popularity of Federated Learning (FL). While there are many existing works focusing on various aspects of the FL process, the issue of the effective problem diagnosis in FL systems remains largely unexplored. In this work, we have set out to artificially simulate the training process of four selected approaches to FL topology and compare their resulting performance. After noticing concerning disturbances throughout their training process, we have successfully identified their source as the problem of exploding gradients. We have then made modifications to the model structure and analyzed the new results. Finally, we have proposed continuous monitoring of the FL training process through the local computation of a selected metric.

Słowa kluczowe

EN

measurement; machine learning algorithm; federated learning; focusing; maintenance engineering; topology

PL

pomiar; algorytmy uczenia maszynowego; uczenie federacyjne; koncentracja; inżynieria; konserwacja; topologia

• Wydawca: Polskie Towarzystwo Informatyczne
• Czasopismo: Annals of Computer Science and Information Systems
• Rocznik: 2023
• Tom: Vol. 35
• Strony: 871--876
• Opis fizyczny: Bibliogr. 25 poz., il., wykr.

Twórcy

autor

Bogacka Karolina

• Warsaw University of Technology Plac Politechniki 1, 00-661 Warszawa, Poland

• https://orcid.org/0000-0002-7109-891X

autor

Danilenka Anastasiya

• Warsaw University of Technology Plac Politechniki 1, 00-661 Warszawa, Poland

• https://orcid.org/0000-0002-3080-0303

autor

Wasielewska-Michniewska Katarzyna

• Systems Research Institute, Polish Academy of Sciences Newelska 6, 01-447 Warszawa, Poland

• https://orcid.org/0000-0002-3763-2373

Bibliografia

• 1. L. U. Khan, W. Saad, Z. Han, E. Hossain, and C. S. Hong, “Federated learning for internet of things: Recent advances, taxonomy, and open challenges,” CoRR, vol. abs/2009.13012, 2020. [Online]. Available: https://arxiv.org/abs/2009.13012
• 2. H. B. McMahan, E. Moore, D. Ramage, and B. A. y Arcas, “Federated learning of deep networks using model averaging,” CoRR, vol. abs/1602.05629, 2016. [Online]. Available: http://arxiv.org/abs/1602.05629
• 3. J. Wu, S. Drew, F. Dong, Z. Zhu, and J. Zhou, “Topology-aware federated learning in edge computing: A comprehensive survey,” 2023. [Online]. Available: https://arxiv.org/abs/2302.02573
• 4. “Pilot Scenario Implementation – First Version,” 2022. [Online]. Available: https://assist-iot.eu/wp-content/uploads/2022/05/D7.2_Pilot_Scenario_Implementation-First_Version.pdf
• 5. Introducing Federated Learning into Internet of Things ecosystems – preliminary considerations, 07 2022.
• 6. G. Philipp, D. Song, and J. G. Carbonell, “The exploding gradient problem demystified - definition, prevalence, impact, origin, tradeoffs, and solutions,” 2018.
• 7. A. Li, L. Zhang, J. Wang, F. Han, and X.-Y. Li, “Privacy-preserving efficient federated-learning model debugging,” IEEE Transactions on Parallel and Distributed Systems, vol. 33, no. 10, pp. 2291–2303, 2022.
• 8. Y. Liu, W. Wu, L. Flokas, J. Wang, and E. Wu, “Enabling sql-based training data debugging for federated learning,” CoRR, vol. abs/2108.11884, 2021. [Online]. Available: https://arxiv.org/abs/2108.11884
• 9. W. Gill, A. Anwar, and M. A. Gulzar, “Feddebug: Systematic debugging for federated learning applications,” 2023.
• 10. S. Duan, C. Liu, P. Han, X. Jin, X. Zhang, X. Xiang, H. Pan et al., “Fed-dnn-debugger: Automatically debugging deep neural network models in federated learning,” Security and Communication Networks, vol. 2023, 2023.
• 11. B. Hanin, “Which neural net architectures give rise to exploding and vanishing gradients?” in Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. CesaBianchi, and R. Garnett, Eds., vol. 31. Curran Associates, Inc., 2018. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2018/file/13f9896df61279c928f19721878fac41-Paper.pdf
• 12. A. Bellet, A. Kermarrec, and E. Lavoie, “D-cliques: Compensating noniidness in decentralized federated learning with topology,” CoRR, vol. abs/2104.07365, 2021. [Online]. Available: https://arxiv.org/abs/2104.07365
• 13. L. Chou, Z. Liu, Z. Wang, and A. Shrivastava, “Efficient and less centralized federated learning,” CoRR, vol. abs/2106.06627, 2021. [Online]. Available: https://arxiv.org/abs/2106.06627
• 14. N. Mhaisen, A. A. Abdellatif, A. Mohamed, A. Erbad, and M. Guizani, “Optimal user-edge assignment in hierarchical federated learning based on statistical properties and network topology constraints,” IEEE Transactions on Network Science and Engineering, vol. 9, no. 1, pp. 55–66, 2022.
• 15. J. Lee, J. Oh, S. Lim, S. Yun, and J. Lee, “Tornadoaggregate: Accurate and scalable federated learning via the ring-based architecture,” CoRR, vol. abs/2012.03214, 2020. [Online]. Available: https://arxiv.org/abs/2012.03214
• 16. I. Hegedűs, G. Danner, and M. Jelasity, “Gossip learning as a decentralized alternative to federated learning,” in Distributed Applications and Interoperable Systems, J. Pereira and L. Ricci, Eds. Cham: Springer International Publishing, 2019, pp. 74–90.
• 17. Y. Shi, Y. E. Sagduyu, and T. Erpek, “Federated learning for distributed spectrum sensing in nextg communication networks,” 2022. [Online]. Available: https://arxiv.org/abs/2204.03027
• 18. H. Eichner, T. Koren, H. B. McMahan, N. Srebro, and K. Talwar, “Semi-cyclic stochastic gradient descent,” CoRR, vol. abs/1904.10120, 2019. [Online]. Available: http://arxiv.org/abs/1904.10120
• 19. A. Ghosh, J. Chung, D. Yin, and K. Ramchandran, “An efficient framework for clustered federated learning,” 2021.
• 20. Harshvardhan, A. Ghosh, and A. Mazumdar, “An improved algorithm for clustered federated learning,” 2022.
• 21. M. Zhang, E. Wei, and R. Berry, “Faithful edge federated learning: Scalability and privacy,” IEEE Journal on Selected Areas in Communications, vol. 39, no. 12, pp. 3790–3804, 2021.
• 22. D. Yin, Y. Chen, R. Kannan, and P. Bartlett, “Byzantine-robust distributed learning: Towards optimal statistical rates,” in Proceedings of the 35th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, J. Dy and A. Krause, Eds., vol. 80. PMLR, 10–15 Jul 2018, pp. 5650–5659. [Online]. Available: https://proceedings.mlr.press/v80/yin18a.html
• 23. J. Wu, S. Drew, F. Dong, Z. Zhu, and J. Zhou, “Topology-aware federated learning in edge computing: A comprehensive survey,” arXiv preprint https://arxiv.org/abs/2302.02573, 2023.
• 24. L. Liu, J. Zhang, S. Song, and K. B. Letaief, “Edge-assisted hierarchical federated learning with non-iid data,” CoRR, vol. abs/1905.06641, 2019. [Online]. Available: http://arxiv.org/abs/1905.06641
• 25. J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel, “Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition,” Neural Networks, vol. 32, pp. 323–332, 2012, selected Papers from IJCNN 2011. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0893608012000457

Uwagi

1. The work of Karolina Bogacka and Anastasiya Danilenka was funded in part by the Centre for Priority Research Area Artificial Intelligence and Robotics of Warsaw University of Technology within the Excellence Initiative: Research University (IDUB) programme.

2. Thematic Tracks Short Papers

3. Opracowanie rekordu ze środków MEiN, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2024).