Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting
For researchers and practitioners using LLMs for probabilistic forecasting, BBC provides a lightweight method to improve calibration and accuracy by incorporating human uncertainty signals.
The paper introduces the Beta-Bernoulli Calibrator (BBC) to improve LLM probabilistic forecasting by leveraging both binary outcomes and human forecast agreement. BBC achieves better calibrated and more accurate forecasts than existing methods, with its epistemic uncertainty serving as a more reliable predictor of forecasting error.
Probabilistic forecasting estimates the likelihood of uncertain future events. To improve LLM forecasting, existing methods typically learn from binary outcomes to output verbalized forecasts. However, while aggregated human forecasts contain rich information in both the crowd probability estimate and the degree of agreement among forecasters, how to utilize these signals remains underexplored. To address this, we propose the Beta-Bernoulli Calibrator (BBC), which converts an initial point estimate forecast from any model into a distribution over event likelihood, using supervision from both binary outcomes and human forecasts. BBC models event likelihood $p \sim \text{Beta}(α, β)$ and outcome $y \sim \text{Bernoulli}(p)$, with the mean as the calibrated point forecast and the variance as the epistemic uncertainty. Our results show that BBC generally provides better calibrated and more accurate forecasts than both traditional post-hoc calibration methods and models fine-tuned specifically for forecasting, while remaining lightweight and having good generalization. We also show that the epistemic uncertainty captured by BBC is a more reliable predictor of forecasting error than verbalized confidence.