Interpretable Probability Estimation with LLMs via Shapley Reconstruction
This addresses the need for interpretable and accurate probability estimation in LLM-based decision support systems, though it is an incremental improvement using existing Shapley methods on new data.
The paper tackles the problem of noisy and opaque probability estimation from LLMs by proposing PRISM, a framework that uses Shapley values to decompose and reconstruct calibrated estimates. The result shows improved predictive accuracy over direct prompting across domains like finance and healthcare.
Large Language Models (LLMs) demonstrate potential to estimate the probability of uncertain events, by leveraging their extensive knowledge and reasoning capabilities. This ability can be applied to support intelligent decision-making across diverse fields, such as financial forecasting and preventive healthcare. However, directly prompting LLMs for probability estimation faces significant challenges: their outputs are often noisy, and the underlying predicting process is opaque. In this paper, we propose PRISM: Probability Reconstruction via Shapley Measures, a framework that brings transparency and precision to LLM-based probability estimation. PRISM decomposes an LLM's prediction by quantifying the marginal contribution of each input factor using Shapley values. These factor-level contributions are then aggregated to reconstruct a calibrated final estimate. In our experiments, we demonstrate PRISM improves predictive accuracy over direct prompting and other baselines, across multiple domains including finance, healthcare, and agriculture. Beyond performance, PRISM provides a transparent prediction pipeline: our case studies visualize how individual factors shape the final estimate, helping build trust in LLM-based decision support systems.