Selective Reincarnation: Offline-to-Online Multi-Agent Reinforcement Learning
This work addresses the challenge of efficient training in multi-agent systems for researchers, but it is incremental as it extends the reincarnation paradigm from single-agent to multi-agent contexts.
The paper tackles the problem of reusing prior computation in multi-agent reinforcement learning by introducing selective reincarnation, where only some agents are reincarnated while others are trained from scratch, and demonstrates that it leads to higher returns than training from scratch and faster convergence than full reincarnation in a fully-cooperative setting with heterogeneous agents.
'Reincarnation' in reinforcement learning has been proposed as a formalisation of reusing prior computation from past experiments when training an agent in an environment. In this paper, we present a brief foray into the paradigm of reincarnation in the multi-agent (MA) context. We consider the case where only some agents are reincarnated, whereas the others are trained from scratch -- selective reincarnation. In the fully-cooperative MA setting with heterogeneous agents, we demonstrate that selective reincarnation can lead to higher returns than training fully from scratch, and faster convergence than training with full reincarnation. However, the choice of which agents to reincarnate in a heterogeneous system is vitally important to the outcome of the training -- in fact, a poor choice can lead to considerably worse results than the alternatives. We argue that a rich field of work exists here, and we hope that our effort catalyses further energy in bringing the topic of reincarnation to the multi-agent realm.