Is Feedback All You Need? Leveraging Natural Language Feedback in Goal-Conditioned Reinforcement Learning
This work addresses the challenge of bridging the gap between RL and human-like learning for AI researchers, though it is incremental as it builds on existing methods like Decision Transformer and BabyAI.
The paper tackles the problem of improving generalization in reinforcement learning by incorporating natural language feedback, showing that agents trained with such feedback achieve better generalization performance, with specific gains observed when feedback is used in place of or alongside traditional signals like return-to-go or goal descriptions.
Despite numerous successes, the field of reinforcement learning (RL) remains far from matching the impressive generalisation power of human behaviour learning. One possible way to help bridge this gap be to provide RL agents with richer, more human-like feedback expressed in natural language. To investigate this idea, we first extend BabyAI to automatically generate language feedback from the environment dynamics and goal condition success. Then, we modify the Decision Transformer architecture to take advantage of this additional signal. We find that training with language feedback either in place of or in addition to the return-to-go or goal descriptions improves agents' generalisation performance, and that agents can benefit from feedback even when this is only available during training, but not at inference.