Optimistic Distributionally Robust Optimization for Nonparametric Likelihood Approximation
This work addresses a fundamental bottleneck in Bayesian inference for statisticians and machine learning practitioners, offering a novel method with theoretical guarantees, though it appears incremental as it builds on existing distributionally robust optimization frameworks.
The paper tackles the problem of computationally intractable likelihood evaluation in Bayesian statistics by proposing a non-parametric approximation called optimistic likelihood, which identifies a probability measure in a neighborhood of the nominal measure to maximize sample probability; it shows that this approximation leads to convex optimization problems with analytical expressions in some cases and performs competitively in a probabilistic classification task.
The likelihood function is a fundamental component in Bayesian statistics. However, evaluating the likelihood of an observation is computationally intractable in many applications. In this paper, we propose a non-parametric approximation of the likelihood that identifies a probability measure which lies in the neighborhood of the nominal measure and that maximizes the probability of observing the given sample point. We show that when the neighborhood is constructed by the Kullback-Leibler divergence, by moment conditions or by the Wasserstein distance, then our \textit{optimistic likelihood} can be determined through the solution of a convex optimization problem, and it admits an analytical expression in particular cases. We also show that the posterior inference problem with our optimistic likelihood approximation enjoys strong theoretical performance guarantees, and it performs competitively in a probabilistic classification task.