Measuring the Completeness of Theories
This provides a tractable measure for researchers in behavioral sciences and machine learning to evaluate theoretical models, though it is incremental as it applies existing ML methods to a new metric.
The paper tackles the problem of quantifying how much predictable variation in data a theory captures, termed 'completeness', using machine learning, and finds considerable variation in existing models across three applications: lottery equivalents, game play, and random sequence generation, with implications for model development and experimental design.
We use machine learning to provide a tractable measure of the amount of predictable variation in the data that a theory captures, which we call its "completeness." We apply this measure to three problems: assigning certain equivalents to lotteries, initial play in games, and human generation of random sequences. We discover considerable variation in the completeness of existing models, which sheds light on whether to focus on developing better models with the same features or instead to look for new features that will improve predictions. We also illustrate how and why completeness varies with the experiments considered, which highlights the role played in choosing which experiments to run.