Query Timing Produces Opposite Positional Biases Between LLMs and Humans
This work provides insights into the underlying mechanisms of positional biases in LLMs, which is important for understanding and mitigating evaluation biases in AI systems, but it is an incremental step in a specific research area.
The paper investigates whether the timing of belief updates (during vs. at the end of evidence presentation) affects positional biases in LLMs, as it does in humans. They find that LLMs exhibit opposite positional biases compared to humans, and these biases are more pronounced in newer models.
Positional biases such as recency and primacy effects have been documented in large language models (LLMs), yet the underlying mechanism by which these models make their evaluations remains poorly understood. Both primacy and recency biases have been observed in human judgments in response to evidence, but recent work suggest that \emph{when} the listener updates their beliefs -- during the presentation of evidence or only at the end -- influences the presence of such effects. We investigate whether a similar phenomenon holds for LLMs, finding divergence from human behavior. These biases are more exacerbated in newer models compared to their predecessors.