Can LLMs Replace Economic Choice Prediction Labs? The Case of Language-based Persuasion Games
This work addresses the difficulty of acquiring human choice data for applications in marketing, finance, and public policy, offering a potential solution through LLM-generated data, though it is incremental in extending existing AI methods to more complex economic contexts.
The paper tackles the problem of predicting human choices in complex economic settings by using LLM-generated data as a substitute for scarce human data, showing that models trained on this data can effectively predict human behavior and even outperform those trained on actual human data in language-based persuasion games.
Human choice prediction in economic contexts is crucial for applications in marketing, finance, public policy, and more. This task, however, is often constrained by the difficulties in acquiring human choice data. With most experimental economics studies focusing on simple choice settings, the AI community has explored whether LLMs can substitute for humans in these predictions and examined more complex experimental economics settings. However, a key question remains: can LLMs generate training data for human choice prediction? We explore this in language-based persuasion games, a complex economic setting involving natural language in strategic interactions. Our experiments show that models trained on LLM-generated data can effectively predict human behavior in these games and even outperform models trained on actual human data. Beyond data generation, we investigate the dual role of LLMs as both data generators and predictors, introducing a comprehensive empirical study on the effectiveness of utilizing LLMs for data generation, human choice prediction, or both. We then utilize our choice prediction framework to analyze how strategic factors shape decision-making, showing that interaction history (rather than linguistic sentiment alone) plays a key role in predicting human decision-making in repeated interactions. Particularly, when LLMs capture history-dependent decision patterns similarly to humans, their predictive success improves substantially. Finally, we demonstrate the robustness of our findings across alternative persuasion-game settings, highlighting the broader potential of using LLM-generated data to model human decision-making.