Instruction Boundary: Quantifying Biases in LLM Reasoning under Various Coverage
This work addresses the problem of inherent flaws in automatically generated datasets for LLM reasoning, which is important for developers to mitigate risks in AI applications, though it is incremental in analyzing prompt effects.
The study investigated how different levels of prompt coverage (sufficient, redundant, or insufficient) cause reasoning biases in LLMs when handling datasets with sparse labels, such as questions with none or multiple correct options, and found that despite high accuracy, substantial biases persist in downstream tasks.
Nowadays, automatically generated datasets are increasingly used in LLM reasoning tasks; however, large-scale corpora often contain inherent flaws. For example, a single-choice question may include none or multiple correct options, while true-or-false questions may involve vague or unverifiable statements. We refer to these exceptional answer forms as sparse labels. To compare LLMs' ability to recognize various question forms and produce correct answers, we investigate how different instruction formats can either facilitate or mislead LLM reasoning ability. We introduce the concept of Instruction Boundary, which systematically analyzes how different levels of prompt coverage -- sufficient, redundant, or insufficient -- can lead to reasoning biases and performance changes in LLMs. To examine this phenomenon, we design eight experimental settings across five dataset forms. We further propose BiasDetector, a unified framework that quantifies LLMs' ability to identify sparse labels under different kinds of Instruction Boundary conditions. Evaluations on five mainstream LLMs show that, despite their seemingly high accuracy, substantial reasoning biases persist in many downstream tasks as a direct consequence of prompt coverage. We analyze the impact of these biases and outline possible mitigation strategies. Our findings highlight not only the importance of addressing sparse labels, but also the need for developers to recognize and mitigate the risks introduced by Instruction Boundary.