FedSteer: Taming Extreme Gradient Staleness in Federated Learning with Corrective Projections and Caching
For federated learning practitioners dealing with non-uniform client participation, FedSteer provides a method to mitigate the destabilizing effects of stale updates, improving model accuracy and stability.
FedSteer addresses the problem of extreme gradient staleness in federated learning caused by skewed client participation. By constructing a gradient subspace from cached recent gradients and projecting stale updates onto it, FedSteer prevents training collapse and achieves accuracy gains of over 7% in challenging scenarios.
Federated learning (FL) is often subject to aggregation variance if clients do not consistently participate in training rounds. While reusing stale model updates from inactive clients is a common technique to reduce this variance, we find that with skewed client participation, the resulting update staleness can become severe enough to destabilize training. To remedy this, we propose FedSteer, a novel method that constructs a gradient subspace from a cache of recent client gradients to serve as a low-dimensional representation of the current optimization landscape. FedSteer projects an active client's true gradient onto this subspace to find a set of optimal coordinates. For an inactive client, FedSteer reuses these coordinates with the now-evolved subspace drifted by other active clients. This process effectively "steers" outdated gradients toward the current global objective. This is complemented by a selective caching strategy that identifies a representative client subset to form the subspace, reducing server memory. Experiments demonstrate that FedSteer significantly outperforms baselines, preventing performance collapse in challenging scenarios while delivering accuracy gains of over 7% in others.