Exploring the Impact of Tunable Agents in Sequential Social Dilemmas
This addresses the inefficiency of retraining agents for behavior changes in social dilemmas, though it is incremental as it builds on existing multi-objective methods.
The study tackled the problem of retraining reinforcement learning agents for new behaviors by developing tunable agents using multi-objective reinforcement learning, enabling adaptation between cooperative and competitive behaviors in sequential social dilemmas without retraining.
When developing reinforcement learning agents, the standard approach is to train an agent to converge to a fixed policy that is as close to optimal as possible for a single fixed reward function. If different agent behaviour is required in the future, an agent trained in this way must normally be either fully or partially retrained, wasting valuable time and resources. In this study, we leverage multi-objective reinforcement learning to create tunable agents, i.e. agents that can adopt a range of different behaviours according to the designer's preferences, without the need for retraining. We apply this technique to sequential social dilemmas, settings where there is inherent tension between individual and collective rationality. Learning a single fixed policy in such settings leaves one at a significant disadvantage if the opponents' strategies change after learning is complete. In our work, we demonstrate empirically that the tunable agents framework allows easy adaption between cooperative and competitive behaviours in sequential social dilemmas without the need for retraining, allowing a single trained agent model to be adjusted to cater for a wide range of behaviours and opponent strategies.