Beyond Linear Attention: Softmax Transformers Implement In-Context Reinforcement Learning
For the RL and ML theory community, this work bridges the gap between practical Transformer-based ICRL and theoretical understanding by removing the unrealistic linear attention assumption.
This paper provides the first theoretical analysis of in-context reinforcement learning (ICRL) using standard softmax attention, showing that a Transformer's forward pass with certain parameters implements a weighted softmax temporal difference (TD) learning algorithm. The authors prove that under a contraction condition, policy evaluation error decays with more layers, and that these parameters are a global minimizer of a pretraining loss.
In-context reinforcement learning (ICRL) studies agents that, after pretraining, adapt to new tasks by conditioning on additional context without parameter updates. Existing theoretical analyses of ICRL largely rely on linear attention, which replaces the softmax function in the standard attention with an identity mapping. This paper provides the first theoretical understanding of ICRL without making the unrealistic linear attention simplification. In particular, we consider the standard softmax attention used in practice. We show that, with certain parameters, the layerwise forward pass of a Transformer with such softmax attention is equivalent to iterative updates of a weighted softmax temporal difference (TD) learning algorithm. Here, weighted softmax TD is a new RL algorithm that performs policy evaluation in kernel space and adopts both linear TD and tabular TD as special cases. We also prove that under a certain contraction condition, the policy evaluation error decays as the number of layers grows, with the identified parameters above. Finally, we prove that those parameters are a global minimizer of a pretraining loss, explaining their emergence in our numerical experiments.