Mitigating Position Bias in Transformers via Layer-Specific Positional Embedding Scaling
For LLM users and developers, LPES offers a practical, inference-time fix for positional bias in long-context tasks, though it is an incremental improvement over existing multi-scale RoPE methods.
LPES mitigates the 'lost-in-the-middle' problem in LLMs by assigning distinct scaling factors to each layer via a genetic algorithm with Bézier curves, achieving up to 11.2% accuracy gain on key-value retrieval without fine-tuning or latency increase.
Large Language Models (LLMs) still struggle with the ``lost-in-the-middle'' problem, where critical information located in the middle of long-context inputs is often underrepresented or lost. While existing methods attempt to address this by combining multi-scale rotary position embeddings (RoPE), they typically suffer from high latency or rely on suboptimal hand-crafted scaling strategies. To overcome these limitations, we introduce a layer-specific positional embedding scaling~(LPES) method that assigns distinct scaling factors to each layer. LPES achieves a more balanced attention distribution without fine-tuning model parameters or increasing inference delay. A specially designed genetic algorithm is employed to efficiently select the optimal scaling factors for each layer by incorporating Bézier curves to significantly reduce the search space. Extensive experiments demonstrate that LPES effectively mitigates positional attention bias and delivers consistent improvements across multiple long-context benchmarks, yielding up to an $11.2$\% accuracy gain on the key-value retrieval dataset.