Experience Sharing Between Cooperative Reinforcement Learning Agents
This addresses the challenge of slow learning in cooperative multiagent systems, though it is incremental as it builds on existing experience sharing ideas with new selection methods.
The paper tackled the problem of accelerating learning in cooperative multiagent reinforcement learning by proposing three new experience sharing methods, showing that Focused ES reduces the number of episodes needed to complete a task by 51% and accelerates learning by a factor of 2 compared to a baseline.
The idea of experience sharing between cooperative agents naturally emerges from our understanding of how humans learn. Our evolution as a species is tightly linked to the ability to exchange learned knowledge with one another. It follows that experience sharing (ES) between autonomous and independent agents could become the key to accelerate learning in cooperative multiagent settings. We investigate if randomly selecting experiences to share can increase the performance of deep reinforcement learning agents, and propose three new methods for selecting experiences to accelerate the learning process. Firstly, we introduce Focused ES, which prioritizes unexplored regions of the state space. Secondly, we present Prioritized ES, in which temporal-difference error is used as a measure of priority. Finally, we devise Focused Prioritized ES, which combines both previous approaches. The methods are empirically validated in a control problem. While sharing randomly selected experiences between two Deep Q-Network agents shows no improvement over a single agent baseline, we show that the proposed ES methods can successfully outperform the baseline. In particular, the Focused ES accelerates learning by a factor of 2, reducing by 51% the number of episodes required to complete the task.