RePo: Language Models with Context Re-Positioning
This work addresses a bottleneck in in-context learning for LLMs, offering a novel method to improve efficiency in handling complex contexts, though it is incremental as it builds on existing models.
The paper tackles the problem of rigid contextual structures in large language models by proposing RePo, a mechanism for context re-positioning based on Cognitive Load Theory, which enhances performance on tasks with noisy contexts, structured data, and longer contexts while maintaining competitiveness on general short-context tasks.
In-context learning is fundamental to modern Large Language Models (LLMs); however, prevailing architectures impose a rigid and fixed contextual structure by assigning linear or constant positional indices. Drawing on Cognitive Load Theory (CLT), we argue that this uninformative structure increases extraneous cognitive load, consuming finite working memory capacity that should be allocated to deep reasoning and attention allocation. To address this, we propose RePo, a novel mechanism that reduces extraneous load via context re-positioning. Unlike standard approaches, RePo utilizes a differentiable module, $f_φ$, to assign token positions that capture contextual dependencies, rather than replying on pre-defined order. By continually pre-training on the OLMo-2 1B & 7B models, we demonstrate that RePo consistently enhances performance on tasks involving noisy contexts, structured data, and longer context length, while maintaining competitive performance on general short-context tasks. Detailed analysis reveals that RePo successfully allocate higher attention to distant but relevant information, assign positions in dense and non-linear space, and capture the intrinsic structure of the input context. We will open-source the code and model weights. Our code is at https://github.com/SakanaAI/repo.