How to Evaluate Behavioral Models
This work addresses a methodological gap for researchers in behavioral modeling, such as game theorists, by providing a principled framework for evaluation, though it is incremental in refining existing practices.
The paper tackles the problem of selecting appropriate loss functions for evaluating behavioral models, proposing a family of diagonal bounded Bregman divergences that satisfy formal axioms and recommending squared L2 error as a suitable choice.
Researchers building behavioral models, such as behavioral game theorists, use experimental data to evaluate predictive models of human behavior. However, there is little agreement about which loss function should be used in evaluations, with error rate, negative log-likelihood, cross-entropy, Brier score, and squared L2 error all being common choices. We attempt to offer a principled answer to the question of which loss functions should be used for this task, formalizing axioms that we argue loss functions should satisfy. We construct a family of loss functions, which we dub "diagonal bounded Bregman divergences", that satisfy all of these axioms. These rule out many loss functions used in practice, but notably include squared L2 error; we thus recommend its use for evaluating behavioral models.