All You Need Is Supervised Learning: From Imitation Learning to Meta-RL With Upside Down RL
This work addresses the problem of fragmented RL methods for researchers and practitioners by proposing a unified approach, though it appears incremental as it extends an existing method to new settings.
The paper tackles the challenge of unifying multiple reinforcement learning paradigms by demonstrating that Upside Down RL (UDRL), which uses supervised learning to predict actions from returns, can be applied to imitation learning, offline RL, goal-conditioned RL, and meta-RL with a single algorithm and architecture.
Upside down reinforcement learning (UDRL) flips the conventional use of the return in the objective function in RL upside down, by taking returns as input and predicting actions. UDRL is based purely on supervised learning, and bypasses some prominent issues in RL: bootstrapping, off-policy corrections, and discount factors. While previous work with UDRL demonstrated it in a traditional online RL setting, here we show that this single algorithm can also work in the imitation learning and offline RL settings, be extended to the goal-conditioned RL setting, and even the meta-RL setting. With a general agent architecture, a single UDRL agent can learn across all paradigms.