Position Bias Correction is Insufficient for One-Pass Attention Sorting
For researchers working on long-context language models, this paper shows that position-bias correction is not enough to replace iterative sorting, highlighting the need for additional mechanisms.
The authors hypothesized that position bias is the main bottleneck in Attention Sorting and proposed Debiased One-Pass Attention Sorting to correct it. However, experiments on two models showed that debiasing either produced identical results to uncalibrated single-pass sorting (94.83% accuracy) or closed only 37% of the gap to iterative sorting, indicating that position-bias correction alone is insufficient.
Long-context language models suffer from position bias, where information in middle positions is underutilized. Attention Sorting addresses this by iteratively reordering documents based on attention patterns, but its multiple sort-and-generate cycles increase deployment cost. We hypothesize that position bias is the primary bottleneck and propose Debiased One-Pass Attention Sorting, which estimates a per-prompt position-bias curve from the low-attention majority of documents and uses it to correct raw attention scores (via subtraction or division) to enable single-pass sorting. Our experiments on two models refute this hypothesis in the tested setting: on LLaMA-2-7B-32K-Instruct, debiasing produces identical results to uncalibrated single-pass sorting (94.83\% containment accuracy), while on YaRN-Llama-2-7b-64k, debiasing improves accuracy by 8.67 percentage points but remains 14.84pp behind iterative sorting, closing only 37\% of the gap. These results suggest that position-bias correction is insufficient to match iterative sorting, and that repeated reordering provides additional benefits beyond bias correction.