The Effect of Training Parameters and Mechanisms on Decentralized Federated Learning based on MNIST Dataset
This work addresses the bottleneck of central servers in federated learning for decentralized data scenarios, but it is incremental as it primarily tests existing methods on a standard dataset.
The paper tackled the problem of decentralized federated learning (DFL) by experimenting with various training parameters and mechanisms on the MNIST dataset, finding that altered procedures are generally robust but non-optimal, with failures occurring when model weight variance is too large.
Federated Learning is an algorithm suited for training models on decentralized data, but the requirement of a central "server" node is a bottleneck. In this document, we first introduce the notion of Decentralized Federated Learning (DFL). We then perform various experiments on different setups, such as changing model aggregation frequency, switching from independent and identically distributed (IID) dataset partitioning to non-IID partitioning with partial global sharing, using different optimization methods across clients, and breaking models into segments with partial sharing. All experiments are run on the MNIST handwritten digits dataset. We observe that those altered training procedures are generally robust, albeit non-optimal. We also observe failures in training when the variance between model weights is too large. The open-source experiment code is accessible through GitHub\footnote{Code was uploaded at \url{https://github.com/zhzhang2018/DecentralizedFL}}.