Predicting human decisions with behavioral theories and machine learning
This work addresses the problem of improving prediction accuracy for human decision-making in fields like psychology and economics, representing a novel integration rather than an incremental step.
The paper tackled the challenge of predicting human decisions under risk and uncertainty by introducing BEAST-GB, a hybrid model combining behavioral theory with machine learning, which won a competition and outperformed neural networks and existing models in accuracy and generalization across datasets.
Predicting human decisions under risk and uncertainty remains a fundamental challenge across disciplines. Existing models often struggle even in highly stylized tasks like choice between lotteries. We introduce BEAST Gradient Boosting (BEAST-GB), a hybrid model integrating behavioral theory (BEAST) with machine learning. We first present CPC18, a competition for predicting risky choice, in which BEAST-GB won. Then, using two large datasets, we demonstrate BEAST-GB predicts more accurately than neural networks trained on extensive data and dozens of existing behavioral models. BEAST-GB also generalizes robustly across unseen experimental contexts, surpassing direct empirical generalization, and helps refine and improve the behavioral theory itself. Our analyses highlight the potential of anchoring predictions on behavioral theory even in data-rich settings and even when the theory alone falters. Our results underscore how integrating machine learning with theoretical frameworks, especially those-like BEAST-designed for prediction, can improve our ability to predict and understand human behavior.