FedNNNN: Norm-Normalized Neural Network Aggregation for Fast and Accurate Federated Learning
This work addresses a specific bottleneck in federated learning for distributed systems, offering an incremental improvement over existing methods like FedAvg.
The paper tackles the problem of slow convergence and degraded accuracy in federated learning by proposing FedNNNN, a norm-normalized aggregation method that adjusts update vector norms and uses momentum control, achieving up to 5.4% accuracy improvement.
Federated learning (FL) is a distributed learning protocol in which a server needs to aggregate a set of models learned some independent clients to proceed the learning process. At present, model averaging, known as FedAvg, is one of the most widely adapted aggregation techniques. However, it is known to yield the models with degraded prediction accuracy and slow convergence. In this work, we find out that averaging models from different clients significantly diminishes the norm of the update vectors, resulting in slow learning rate and low prediction accuracy. Therefore, we propose a new aggregation method called FedNNNN. Instead of simple model averaging, we adjust the norm of the update vector and introduce momentum control techniques to improve the aggregation effectiveness of FL. As a demonstration, we evaluate FedNNNN on multiple datasets and scenarios with different neural network models, and observe up to 5.4% accuracy improvement.