A Generative Bayesian Model for Aggregating Experts' Probabilities
This work addresses the challenge of combining expert probabilities for decision-makers in low-data scenarios, representing an incremental improvement over existing aggregation methods.
The paper tackles the problem of aggregating probability forecasts from multiple experts or classifiers when training data is scarce, by developing a generative Bayesian model that incorporates prior knowledge and expert properties, resulting in a weighted logarithmic opinion pool that meets consistency criteria and shows improved accuracy in empirical tests.
In order to improve forecasts, a decisionmaker often combines probabilities given by various sources, such as human experts and machine learning classifiers. When few training data are available, aggregation can be improved by incorporating prior knowledge about the event being forecasted and about salient properties of the experts. To this end, we develop a generative Bayesian aggregation model for probabilistic classi cation. The model includes an event-specific prior, measures of individual experts' bias, calibration, accuracy, and a measure of dependence betweeen experts. Rather than require absolute measures, we show that aggregation may be expressed in terms of relative accuracy between experts. The model results in a weighted logarithmic opinion pool (LogOps) that satis es consistency criteria such as the external Bayesian property. We derive analytic solutions for independent and for exchangeable experts. Empirical tests demonstrate the model's use, comparing its accuracy with other aggregation methods.