Learning How to Infer Partial MDPs for In-Context Adaptation and Exploration
This addresses the challenge of efficient generalization across tasks for agents, though it is incremental as it builds on existing posterior sampling methods.
The paper tackles the problem of in-context adaptation and exploration in reinforcement learning by using a transformer to learn partial Markov decision processes from training tasks, achieving adaptation speed and exploration-exploitation balance comparable to an exact posterior sampling oracle in the Symbolic Alchemy benchmark.
To generalize across tasks, an agent should acquire knowledge from past tasks that facilitate adaptation and exploration in future tasks. We focus on the problem of in-context adaptation and exploration, where an agent only relies on context, i.e., history of states, actions and/or rewards, rather than gradient-based updates. Posterior sampling (extension of Thompson sampling) is a promising approach, but it requires Bayesian inference and dynamic programming, which often involve unknowns (e.g., a prior) and costly computations. To address these difficulties, we use a transformer to learn an inference process from training tasks and consider a hypothesis space of partial models, represented as small Markov decision processes that are cheap for dynamic programming. In our version of the Symbolic Alchemy benchmark, our method's adaptation speed and exploration-exploitation balance approach those of an exact posterior sampling oracle. We also show that even though partial models exclude relevant information from the environment, they can nevertheless lead to good policies.