State Soup: In-Context Skill Learning, Retrieval and Mixing
This work addresses the challenge of fast model adaptation for sequence modeling, though it appears incremental as it builds on existing methods like parameter interpolation and in-context learning.
The paper tackled the problem of efficiently merging and adapting pretrained recurrent models by proposing to treat internal states as task vectors that can be stored, retrieved, and linearly combined, resulting in improved next-token perplexity and downstream in-context learning task performance.
A new breed of gated-linear recurrent neural networks has reached state-of-the-art performance on a range of sequence modeling problems. Such models naturally handle long sequences efficiently, as the cost of processing a new input is independent of sequence length. Here, we explore another advantage of these stateful sequence models, inspired by the success of model merging through parameter interpolation. Building on parallels between fine-tuning and in-context learning, we investigate whether we can treat internal states as task vectors that can be stored, retrieved, and then linearly combined, exploiting the linearity of recurrence. We study this form of fast model merging on Mamba-2.8b, a pretrained recurrent model, and present preliminary evidence that simple linear state interpolation methods suffice to improve next-token perplexity as well as downstream in-context learning task performance.