RL + Transformer = A General-Purpose Problem Solver
This work addresses the challenge of creating general-purpose problem solvers for AI applications, presenting a novel approach with broad potential impact, though it appears incremental as it builds on existing transformer and RL methods.
The paper tackles the problem of enabling AI to meta-learn and solve new problems beyond its training, demonstrating that a pre-trained transformer fine-tuned with reinforcement learning achieves In-Context Reinforcement Learning (ICRL), solving unseen in-distribution environments with remarkable sample efficiency and performing strongly in out-of-distribution environments.
What if artificial intelligence could not only solve problems for which it was trained but also learn to teach itself to solve new problems (i.e., meta-learn)? In this study, we demonstrate that a pre-trained transformer fine-tuned with reinforcement learning over multiple episodes develops the ability to solve problems that it has never encountered before - an emergent ability called In-Context Reinforcement Learning (ICRL). This powerful meta-learner not only excels in solving unseen in-distribution environments with remarkable sample efficiency, but also shows strong performance in out-of-distribution environments. In addition, we show that it exhibits robustness to the quality of its training data, seamlessly stitches together behaviors from its context, and adapts to non-stationary environments. These behaviors demonstrate that an RL-trained transformer can iteratively improve upon its own solutions, making it an excellent general-purpose problem solver.