Federated Dynamical Low-Rank Training with Global Loss Convergence Guarantees
This addresses performance bottlenecks for clients in federated learning, though it is incremental as it builds on existing low-rank optimization methods.
The paper tackled the problem of high client compute and communication costs in federated learning by proposing a federated dynamical low-rank training scheme, which reduced these costs by up to an order of magnitude with minimal accuracy loss in computer vision benchmarks.
In this work, we propose a federated dynamical low-rank training (FeDLRT) scheme to reduce client compute and communication costs - two significant performance bottlenecks in horizontal federated learning. Our method builds upon dynamical low-rank splitting schemes for manifold-constrained optimization to create a global low-rank basis of network weights, which enables client training on a small coefficient matrix. A consistent global low-rank basis allows us to incorporate a variance correction scheme and prove global loss descent and convergence to a stationary point. Dynamic augmentation and truncation of the low-rank bases automatically optimizes computing and communication resource utilization. We demonstrate the efficiency of FeDLRT in an array of computer vision benchmarks and show a reduction of client compute and communication costs by up to an order of magnitude with minimal impacts on global accuracy.