Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformers
This addresses the need for interpretable and parameter-free learning in large language models, though it is incremental as it builds on existing ICL and transformer frameworks.
The paper tackles the problem of making In-Context Learning (ICL) explicit and permanent in linearized-attention transformers by converting ICL demonstration prompts into bias terms, achieving exact conversion with an inexpensive algorithm (ICLCA) and demonstrating efficacy through experiments.
In-Context Learning (ICL) has been a powerful emergent property of large language models that has attracted increasing attention in recent years. In contrast to regular gradient-based learning, ICL is highly interpretable and does not require parameter updates. In this paper, we show that, for linearized transformer networks, ICL can be made explicit and permanent through the inclusion of bias terms. We mathematically demonstrate the equivalence between a model with ICL demonstration prompts and the same model with the additional bias terms. Our algorithm (ICLCA) allows for exact conversion in an inexpensive manner. Existing methods are not exact and require expensive parameter updates. We demonstrate the efficacy of our approach through experiments that show the exact incorporation of ICL tokens into a linear transformer. We further suggest how our method can be adapted to achieve cheap approximate conversion of ICL tokens, even in regular transformer networks that are not linearized. Our experiments on GPT-2 show that, even though the conversion is only approximate, the model still gains valuable context from the included bias terms.