Unveiling Selection Biases: Exploring Order and Token Sensitivity in Large Language Models
This addresses biases in LLM decision-making for selection problems, which is incremental as it builds on existing research on model biases.
The paper investigates selection biases in Large Language Models (LLMs), specifically option order and token sensitivity, and quantifies their impact through empirical analysis across multiple models and tasks, proposing mitigation strategies to enhance model robustness.
In this paper, we investigate the phenomena of "selection biases" in Large Language Models (LLMs), focusing on problems where models are tasked with choosing the optimal option from an ordered sequence. We delve into biases related to option order and token usage, which significantly impact LLMs' decision-making processes. We also quantify the impact of these biases through an extensive empirical analysis across multiple models and tasks. Furthermore, we propose mitigation strategies to enhance model performance. Our key contributions are threefold: 1) Precisely quantifying the influence of option order and token on LLMs, 2) Developing strategies to mitigate the impact of token and order sensitivity to enhance robustness, and 3) Offering a detailed analysis of sensitivity across models and tasks, which informs the creation of more stable and reliable LLM applications for selection problems.