In-Context Compositional Q-Learning for Offline Reinforcement Learning
This addresses the challenge of compositional task handling in offline RL for researchers and practitioners, offering a novel framework with empirical gains.
The paper tackled the problem of accurately estimating Q-functions in offline reinforcement learning by proposing In-context Compositional Q-Learning (ICQL), which formulates Q-learning as a contextual inference problem, resulting in performance improvements of up to 16.4% in kitchen tasks and up to 8.6% and 6.3% in Gym and Adroit tasks.
Accurately estimating the Q-function is a central challenge in offline reinforcement learning. However, existing approaches often rely on a single global Q-function, which struggles to capture the compositional nature of tasks involving diverse subtasks. We propose In-context Compositional Q-Learning (\texttt{ICQL}), the first offline RL framework that formulates Q-learning as a contextual inference problem, using linear Transformers to adaptively infer local Q-functions from retrieved transitions without explicit subtask labels. Theoretically, we show that under two assumptions--linear approximability of the local Q-function and accurate weight inference from retrieved context--\texttt{ICQL} achieves bounded Q-function approximation error, and supports near-optimal policy extraction. Empirically, \texttt{ICQL} substantially improves performance in offline settings: improving performance in kitchen tasks by up to 16.4\%, and in Gym and Adroit tasks by up to 8.6\% and 6.3\%. These results highlight the underexplored potential of in-context learning for robust and compositional value estimation, positioning \texttt{ICQL} as a principled and effective framework for offline RL.