Collaborative Value Function Estimation Under Model Mismatch: A Federated Temporal Difference Analysis
This addresses the challenge of collaborative learning in real-world applications like multi-robot teams where agents face different dynamics, but it is incremental as it builds on existing FedRL methods.
The paper tackles the problem of model mismatch in federated reinforcement learning, where agents operate in different environments, by analyzing federated temporal difference learning (FedTD(0)) and showing that information sharing reduces errors, with empirical results indicating significant mitigation of environment-specific errors.
Federated reinforcement learning (FedRL) enables collaborative learning while preserving data privacy by preventing direct data exchange between agents. However, many existing FedRL algorithms assume that all agents operate in identical environments, which is often unrealistic. In real-world applications, such as multi-robot teams, crowdsourced systems, and large-scale sensor networks, each agent may experience slightly different transition dynamics, leading to inherent model mismatches. In this paper, we first establish linear convergence guarantees for single-agent temporal difference learning (TD(0)) in policy evaluation and demonstrate that under a perturbed environment, the agent suffers a systematic bias that prevents accurate estimation of the true value function. This result holds under both i.i.d. and Markovian sampling regimes. We then extend our analysis to the federated TD(0) (FedTD(0)) setting, where multiple agents, each interacting with its own perturbed environment, periodically share value estimates to collaboratively approximate the true value function of a common underlying model. Our theoretical results indicate the impact of model mismatch, network connectivity, and mixing behavior on the convergence of FedTD(0). Empirical experiments corroborate our theoretical gains, highlighting that even moderate levels of information sharing significantly mitigate environment-specific errors.