Large Language Bayes
This addresses the challenge for domain experts who need Bayesian analysis but find formal modeling difficult, though it appears incremental as it combines existing techniques.
The paper tackles the problem of domain experts lacking time or expertise to write formal Bayesian models by using a large language model and probabilistic programming to generate models from informal descriptions, enabling sensible predictions without manual model specification.
Many domain experts do not have the time or expertise to write formal Bayesian models. This paper takes an informal problem description as input, and combines a large language model and a probabilistic programming language to define a joint distribution over formal models, latent variables, and data. A posterior over latent variables follows by conditioning on observed data and integrating over formal models. This presents a challenging inference problem. We suggest an inference recipe that amounts to generating many formal models from the large language model, performing approximate inference on each, and then doing a weighted average. This is justified and analyzed as a combination of self-normalized importance sampling, MCMC, and importance-weighted variational inference. Experimentally, this produces sensible predictions from only data and an informal problem description, without the need to specify a formal model.