Optimal design of experiments to identify latent behavioral types
This work improves experimental efficiency for behavioral scientists by reducing data needs, though it is incremental as it builds on an existing method.
The authors tackled the problem of efficiently identifying behavioral models by extending a Bayesian optimal experiment design method with computational improvements, finding that their approach requires significantly less data than expert-chosen designs to distinguish models, with reinforcement learning best explaining human decision-making in an imperfect information game.
Bayesian optimal experiments that maximize the information gained from collected data are critical to efficiently identify behavioral models. We extend a seminal method for designing Bayesian optimal experiments by introducing two computational improvements that make the procedure tractable: (1) a search algorithm from artificial intelligence that efficiently explores the space of possible design parameters, and (2) a sampling procedure which evaluates each design parameter combination more efficiently. We apply our procedure to a game of imperfect information to evaluate and quantify the computational improvements. We then collect data across five different experimental designs to compare the ability of the optimal experimental design to discriminate among competing behavioral models against the experimental designs chosen by a "wisdom of experts" prediction experiment. We find that data from the experiment suggested by the optimal design approach requires significantly less data to distinguish behavioral models (i.e., test hypotheses) than data from the experiment suggested by experts. Substantively, we find that reinforcement learning best explains human decision-making in the imperfect information game and that behavior is not adequately described by the Bayesian Nash equilibrium. Our procedure is general and computationally efficient and can be applied to dynamically optimize online experiments.