Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs
This work addresses a key limitation in LLMs for real-world applications involving long inputs, though it is incremental as it builds on prior research on positional bias.
The paper tackles the problem of positional bias in long-context LLMs, particularly when multiple relevant information pieces are involved, revealing that while models are robust against the 'lost in the middle' issue, significant biases exist related to the spacing of information pieces, as shown through experiments with 11 models on the LongPiBench benchmark.
Positional bias in large language models (LLMs) hinders their ability to effectively process long inputs. A prominent example is the "lost in the middle" phenomenon, where LLMs struggle to utilize relevant information situated in the middle of the input. While prior research primarily focuses on single pieces of relevant information, real-world applications often involve multiple relevant information pieces. To bridge this gap, we present LongPiBench, a benchmark designed to assess positional bias involving multiple pieces of relevant information. Thorough experiments are conducted with five commercial and six open-source models. These experiments reveal that while most current models are robust against the "lost in the middle" issue, there exist significant biases related to the spacing of relevant information pieces. These findings highlight the importance of evaluating and reducing positional biases to advance LLM's capabilities.