Convergence Guarantees for Federated SARSA with Local Training and Heterogeneous Agents
This work addresses convergence issues in federated reinforcement learning for heterogeneous agents, offering theoretical guarantees that could benefit distributed AI systems, though it appears incremental as it builds on existing methods.
The paper tackles the problem of analyzing Federated SARSA with linear function approximation and local training under heterogeneous agents, establishing convergence guarantees and providing the first sample and communication complexity bounds in this setting, with numerical experiments supporting the findings.
We present a novel theoretical analysis of Federated SARSA (FedSARSA) with linear function approximation and local training. We establish convergence guarantees for FedSARSA in the presence of heterogeneity, both in local transitions and rewards, providing the first sample and communication complexity bounds in this setting. At the core of our analysis is a new, exact multi-step error expansion for single-agent SARSA, which is of independent interest. Our analysis precisely quantifies the impact of heterogeneity, demonstrating the convergence of FedSARSA with multiple local updates. Crucially, we show that FedSARSA achieves linear speed-up with respect to the number of agents, up to higher-order terms due to Markovian sampling. Numerical experiments support our theoretical findings.