Mitigating Goal Misgeneralization via Minimax Regret
This addresses a safety issue in AI systems by proposing a theoretical and empirical approach to prevent unintended behaviors during deployment, though it is incremental as it builds on existing regret-based methods.
The paper tackles the problem of goal misgeneralization in reinforcement learning, where policies pursue proxy goals instead of intended ones in deployment, and shows that minimax expected regret objectives are more robust than maximum expected value, with empirical validation in grid-world environments.
Safe generalization in reinforcement learning requires not only that a learned policy acts capably in new situations, but also that it uses its capabilities towards the pursuit of the designer's intended goal. The latter requirement may fail when a proxy goal incentivizes similar behavior to the intended goal within the training environment, but not in novel deployment environments. This creates the risk that policies will behave as if in pursuit of the proxy goal, rather than the intended goal, in deployment -- a phenomenon known as goal misgeneralization. In this paper, we formalize this problem setting in order to theoretically study the possibility of goal misgeneralization under different training objectives. We show that goal misgeneralization is possible under approximate optimization of the maximum expected value (MEV) objective, but not the minimax expected regret (MMER) objective. We then empirically show that the standard MEV-based training method of domain randomization exhibits goal misgeneralization in procedurally-generated grid-world environments, whereas current regret-based unsupervised environment design (UED) methods are more robust to goal misgeneralization (though they don't find MMER policies in all cases). Our findings suggest that minimax expected regret is a promising approach to mitigating goal misgeneralization.