Unexpectedly Useful: Convergence Bounds And Real-World Distributed Learning
This work addresses improving federated learning efficiency by leveraging theoretical bounds, though it is incremental in applying existing methods to new insights.
The study investigated how convergence bounds predict and improve real-world federated learning performance, finding that while bounds are loose and reflect training loss, certain bound quantities unexpectedly help identify beneficial clients without disclosing dataset details.
Convergence bounds are one of the main tools to obtain information on the performance of a distributed machine learning task, before running the task itself. In this work, we perform a set of experiments to assess to which extent, and in which way, such bounds can predict and improve the performance of real-world distributed (namely, federated) learning tasks. We find that, as can be expected given the way they are obtained, bounds are quite loose and their relative magnitude reflects the training rather than the testing loss. More unexpectedly, we find that some of the quantities appearing in the bounds turn out to be very useful to identify the clients that are most likely to contribute to the learning process, without requiring the disclosure of any information about the quality or size of their datasets. This suggests that further research is warranted on the ways -- often counter-intuitive -- in which convergence bounds can be exploited to improve the performance of real-world distributed learning tasks.