Lost in the Middle: An Emergent Property from Information Retrieval Demands in LLMs
This addresses a performance issue in LLMs for AI researchers and developers, though it is incremental as it builds on known biases like primacy and recency effects.
The study tackled the 'lost-in-the-middle' phenomenon in LLMs, where performance drops when key information is in the middle of long contexts, by showing it emerges from training on tasks with different information retrieval demands, such as long-term and short-term memory paradigms, and generalizes to sequence completion tasks.
The performance of Large Language Models (LLMs) often degrades when crucial information is in the middle of a long context, a "lost-in-the-middle" phenomenon that mirrors the primacy and recency effects in human memory. We propose that this behavior is not simply a flaw indicative of information loss but an adaptation to different information retrieval demands during pre-training: some tasks require uniform recall across the entire input (a long-term memory demand), while others prioritize the most recent information (a short-term memory demand). Consistent with this view, we show that this U-shaped performance curve emerges when LLMs (GPT-2 and Llama variants) are trained from scratch on two simple human memory paradigms simulating long-term and short-term memory demands. Our analysis reveals that while the recency effect directly aligns with short-term memory demand in the training data, the primacy effect is induced by the uniform long-term memory demand and is additionally influenced by the model's autoregressive properties and the formation of attention sinks. Our main findings from simple human memory paradigms also generalize to a sequence completion task, which more closely resembles the next-token prediction process in LLM pre-training. Together, our findings reveal how information retrieval demands, model architecture, and structural attention dynamics during model training can jointly produce positional bias observed in LLMs.