Unbiased Gradient Estimation for Distributionally Robust Learning
This work addresses the problem of improving model generalization for machine learning practitioners by proposing a more efficient and robust learning approach.
This paper proposes a new distributionally robust learning (DRL) approach that uses stochastic gradient descent for the outer minimization problem. It efficiently estimates the gradient of the inner maximization problem using multi-level Monte Carlo randomization, leading to significant benefits over prior methods.
Seeking to improve model generalization, we consider a new approach based on distributionally robust learning (DRL) that applies stochastic gradient descent to the outer minimization problem. Our algorithm efficiently estimates the gradient of the inner maximization problem through multi-level Monte Carlo randomization. Leveraging theoretical results that shed light on why standard gradient estimators fail, we establish the optimal parameterization of the gradient estimators of our approach that balances a fundamental tradeoff between computation time and statistical variance. Numerical experiments demonstrate that our DRL approach yields significant benefits over previous work.