The Illusion of Stochasticity in LLMs
This identifies a critical failure point for LLMs in agentic applications, which is incremental but important for reliability.
The paper demonstrates that Large Language Models (LLMs) fail to reliably sample from probability distributions when acting as agents, despite being able to convert random seeds to distributions, with this failure shown empirically across various models and conditions.
In this work, we demonstrate that reliable stochastic sampling is a fundamental yet unfulfilled requirement for Large Language Models (LLMs) operating as agents. Agentic systems are frequently required to sample from distributions, often inferred from observed data, a process which needs to be emulated by the LLM. This leads to a distinct failure point: while standard RL agents rely on external sampling mechanisms, LLMs fail to map their internal probability estimates to their stochastic outputs. Through rigorous empirical analysis across multiple model families, model sizes, prompting styles, and distributions, we demonstrate the extent of this failure. Crucially, we show that while powerful frontier models can convert provided random seeds to target distributions, their ability to sample directly from specific distributions is fundamentally flawed.