Calculating Optimistic Likelihoods Using (Geodesically) Convex Optimization
This addresses the issue of estimation errors in likelihood evaluation for machine learning practitioners, though it appears incremental as it builds on existing ambiguity set and optimization techniques.
The paper tackles the problem of evaluating likelihoods under nominal distributions that are estimated from data and thus prone to errors, by proposing to compute optimistic likelihoods as the maximum likelihood over ambiguity sets around these distributions. The result is an efficient method using convex optimization, demonstrated on a classification problem with synthetic and empirical data.
A fundamental problem arising in many areas of machine learning is the evaluation of the likelihood of a given observation under different nominal distributions. Frequently, these nominal distributions are themselves estimated from data, which makes them susceptible to estimation errors. We thus propose to replace each nominal distribution with an ambiguity set containing all distributions in its vicinity and to evaluate an \emph{optimistic likelihood}, that is, the maximum of the likelihood over all distributions in the ambiguity set. When the proximity of distributions is quantified by the Fisher-Rao distance or the Kullback-Leibler divergence, the emerging optimistic likelihoods can be computed efficiently using either geodesic or standard convex optimization techniques. We showcase the advantages of working with optimistic likelihoods on a classification problem using synthetic as well as empirical data.