Credal Model Averaging for classification: representing prior ignorance and expert opinions
This addresses model uncertainty in classification for domains like ecological prediction, offering a robust alternative to BMA, though it is incremental as it builds on existing ensemble methods.
The paper tackles the sensitivity of Bayesian Model Averaging (BMA) to prior choices on small datasets by introducing Credal Model Averaging (CMA), which uses a set of priors to automate sensitivity analysis and suspend judgment on prior-dependent instances, finding that BMA performs nearly randomly on such instances in a real-world case study.
Bayesian model averaging (BMA) is the state of the art approach for overcoming model uncertainty. Yet, especially on small data sets, the results yielded by BMA might be sensitive to the prior over the models. Credal Model Averaging (CMA) addresses this problem by substituting the single prior over the models by a set of priors (credal set). Such approach solves the problem of how to choose the prior over the models and automates sensitivity analysis. We discuss various CMA algorithms for building an ensemble of logistic regressors characterized by different sets of covariates. We show how CMA can be appropriately tuned to the case in which one is prior-ignorant and to the case in which instead domain knowledge is available. CMA detects prior-dependent instances, namely instances in which a different class is more probable depending on the prior over the models. On such instances CMA suspends the judgment, returning multiple classes. We thoroughly compare different BMA and CMA variants on a real case study, predicting presence of Alpine marmot burrows in an Alpine valley. We find that BMA is almost a random guesser on the instances recognized as prior-dependent by CMA.