Do LLMs Act Like Rational Agents? Measuring Belief Coherence in Probabilistic Decision Making
This addresses the problem of ensuring reliable and interpretable decision-making by LLMs in high-stakes applications like medical diagnosis, though it is incremental in evaluating existing models rather than proposing new methods.
The paper investigates whether large language models (LLMs) act as rational agents with coherent beliefs and stable preferences in probabilistic decision-making, particularly in medical diagnostic domains, by testing them against falsifiable conditions for Bayesian utility maximization.
Large language models (LLMs) are increasingly deployed as agents in high-stakes domains where optimal actions depend on both uncertainty about the world and consideration of utilities of different outcomes, yet their decision logic remains difficult to interpret. We study whether LLMs are rational utility maximizers with coherent beliefs and stable preferences. We consider behaviors of models for diagnosis challenge problems. The results provide insights about the relationship of LLM inferences to ideal Bayesian utility maximization for elicited probabilities and observed actions. Our approach provides falsifiable conditions under which the reported probabilities \emph{cannot} correspond to the true beliefs of any rational agent. We apply this methodology to multiple medical diagnostic domains with evaluations across several LLMs. We discuss implications of the results and directions forward for uses of LLMs in guiding high-stakes decisions.