Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problems
This reveals a critical limitation for researchers and practitioners using LLMs on long-context tasks like document analysis or extended reasoning.
The paper challenges the assumption that self-consistency improves LLM performance on long-context problems, finding it actively degrades performance due to persistent position bias, with degradation worsening with longer contexts and smaller models.
Self-consistency (SC) improves the performance of large language models (LLMs) across various tasks and domains that involve short content. However, does this support its effectiveness for long-context problems? We challenge the assumption that SC's benefits generalize to long-context settings, where LLMs often struggle with position bias, the systematic over-reliance on specific context regions-which hinders their ability to utilize information effectively from all parts of their context. Through comprehensive experimentation with varying state-of-the-art models, tasks, and SC formulations, we find that SC not only fails to improve but actively degrades performance on long-context tasks. This degradation is driven by persistent position bias, which worsens with longer context lengths and smaller model sizes but remains invariant to prompt format or task type. Unlike short-context tasks, where SC diversifies reasoning paths, long-context SC amplifies positional errors. These comprehensive results provide valuable insight into the limitations of current LLMs in long-context understanding and highlight the need for more sophisticated approaches.