Coordinated Exploration in Concurrent Reinforcement Learning
This addresses the challenge of scaling reinforcement learning to multi-agent systems, though it appears incremental as it builds on existing posterior sampling methods.
The paper tackled the problem of efficient coordinated exploration in concurrent reinforcement learning by identifying three necessary properties and proposing seed sampling as a solution, with simulation results showing substantial advantages in per-agent regret reduction as the number of agents increases.
We consider a team of reinforcement learning agents that concurrently learn to operate in a common environment. We identify three properties - adaptivity, commitment, and diversity - which are necessary for efficient coordinated exploration and demonstrate that straightforward extensions to single-agent optimistic and posterior sampling approaches fail to satisfy them. As an alternative, we propose seed sampling, which extends posterior sampling in a manner that meets these requirements. Simulation results investigate how per-agent regret decreases as the number of agents grows, establishing substantial advantages of seed sampling over alternative exploration schemes.