Bayesian Nonparametrics Meets Data-Driven Distributionally Robust Optimization
This work addresses distributional uncertainty in model training for machine learning practitioners, offering a robust optimization approach with theoretical backing, though it appears incremental as it builds on existing distributionally robust optimization and Bayesian nonparametric methods.
The authors tackled the problem of poor out-of-sample performance in machine learning due to distributional uncertainty by proposing a novel robust criterion combining Bayesian nonparametrics and ambiguity-averse preferences, demonstrating favorable finite-sample and asymptotic statistical guarantees.
Training machine learning and statistical models often involves optimizing a data-driven risk criterion. The risk is usually computed with respect to the empirical data distribution, but this may result in poor and unstable out-of-sample performance due to distributional uncertainty. In the spirit of distributionally robust optimization, we propose a novel robust criterion by combining insights from Bayesian nonparametric (i.e., Dirichlet process) theory and a recent decision-theoretic model of smooth ambiguity-averse preferences. First, we highlight novel connections with standard regularized empirical risk minimization techniques, among which Ridge and LASSO regressions. Then, we theoretically demonstrate the existence of favorable finite-sample and asymptotic statistical guarantees on the performance of the robust optimization procedure. For practical implementation, we propose and study tractable approximations of the criterion based on well-known Dirichlet process representations. We also show that the smoothness of the criterion naturally leads to standard gradient-based numerical optimization. Finally, we provide insights into the workings of our method by applying it to a variety of tasks based on simulated and real datasets.