Failure to Mix: Large language models struggle to answer according to desired probability distributions
This reveals a critical limitation in LLMs for applications requiring probabilistic exploration, such as scientific idea generation, and is incremental in highlighting a specific failure mode.
The study found that large language models (LLMs) fail to generate outputs according to specified probability distributions, such as producing '1' 49% of the time resulting in '0' nearly 100% of the time, due to training on benchmarks with correct answers that discourage probabilistic exploration.
Scientific idea generation and selection requires exploration following a target probability distribution. In contrast, current AI benchmarks have objectively correct answers, and training large language models (LLMs) via reinforcement learning against these benchmarks discourages probabilistic exploration. Here, we conducted systematic experiments requesting LLMs to produce outputs following simple probabilistic distributions, and found that all modern LLMs tested grossly fail to follow the distributions. For example, requesting a binary output of "1" 49% of the time produces an answer of "0" nearly 100% of the time. This step function-like behavior of near-exclusively generating the output with marginally highest probability even overrules even strong in-built LLM biases.