Prospect Theory Fails for LLMs: Revealing Instability of Decision-Making under Epistemic Uncertainty
This work addresses the problem of understanding LLM decision-making instability for researchers in AI and behavioral economics, though it is incremental in testing existing theories on new models.
The study investigated whether Prospect Theory, a model of human decision-making under uncertainty, applies to Large Language Models (LLMs) and how epistemic markers (e.g., 'maybe') affect their decisions, finding that modeling LLMs with Prospect Theory is not consistently reliable, especially with diverse linguistic expressions of uncertainty.
Prospect Theory (PT) models human decision-making under uncertainty, while epistemic markers (e.g., maybe) serve to express uncertainty in language. However, it remains largely unexplored whether Prospect Theory applies to contemporary Large Language Models and whether epistemic markers, which express human uncertainty, affect their decision-making behaviour. To address these research gaps, we design a three-stage experiment based on economic questionnaires. We propose a more general and precise evaluation framework to model LLMs' decision-making behaviour under PT, introducing uncertainty through the empirical probability values associated with commonly used epistemic markers in comparable contexts. We then incorporate epistemic markers into the evaluation framework based on their corresponding probability values to examine their influence on LLM decision-making behaviours. Our findings suggest that modelling LLMs' decision-making with PT is not consistently reliable, particularly when uncertainty is expressed in diverse linguistic forms. Our code is released in https://github.com/HKUST-KnowComp/MarPT.