Mind the (DH) Gap! A Contrast in Risky Choices Between Reasoning and Conversational LLMs
This addresses the problem of understanding LLM decision-making for developers and users in AI applications, but it is incremental as it builds on existing comparative studies.
The study compared risky decision-making in 20 large language models (LLMs) under uncertainty, finding that reasoning models (RMs) behave rationally and consistently, while conversational models (CMs) are less rational, more human-like, and sensitive to factors like framing and explanations.
The use of large language models either as decision support systems, or in agentic workflows, is rapidly transforming the digital ecosystem. However, the understanding of LLM decision-making under uncertainty remains limited. We initiate a comparative study of LLM risky choices along two dimensions: (1) prospect representation (explicit vs. experience based) and (2) decision rationale (explanation). Our study, which involves 20 frontier and open LLMs, is complemented by a matched human subjects experiment, which provides one reference point, while an expected payoff maximizing rational agent model provides another. We find that LLMs cluster into two categories: reasoning models (RMs) and conversational models (CMs). RMs tend towards rational behavior, are insensitive to the order of prospects, gain/loss framing, and explanations, and behave similarly whether prospects are explicit or presented via experience history. CMs are significantly less rational, slightly more human-like, sensitive to prospect ordering, framing, and explanation, and exhibit a large description-history gap. Paired comparisons of open LLMs suggest that a key factor differentiating RMs and CMs is training for mathematical reasoning.