Models and Selection Criteria for Regression and Classification
This work addresses model selection issues for researchers and practitioners in Bayesian statistics and machine learning, but it is incremental as it builds on existing criteria and frameworks.
The paper tackles the problem of selecting appropriate Bayesian models for regression and classification by examining a special class of models (BRC) and arguing that transforming arbitrary models into this form can be inappropriate due to ignored prior knowledge. It discusses and contrasts two Bayesian model selection criteria, providing conditions under which they agree.
When performing regression or classification, we are interested in the conditional probability distribution for an outcome or class variable Y given a set of explanatoryor input variables X. We consider Bayesian models for this task. In particular, we examine a special class of models, which we call Bayesian regression/classification (BRC) models, that can be factored into independent conditional (y|x) and input (x) models. These models are convenient, because the conditional model (the portion of the full model that we care about) can be analyzed by itself. We examine the practice of transforming arbitrary Bayesian models to BRC models, and argue that this practice is often inappropriate because it ignores prior knowledge that may be important for learning. In addition, we examine Bayesian methods for learning models from data. We discuss two criteria for Bayesian model selection that are appropriate for repression/classification: one described by Spiegelhalter et al. (1993), and another by Buntine (1993). We contrast these two criteria using the prequential framework of Dawid (1984), and give sufficient conditions under which the criteria agree.