A Model-Based Method for Minimizing CVaR and Beyond
This work addresses the need for more robust model training in machine learning by improving optimization methods for CVaR, though it appears incremental as it builds on existing stochastic prox-linear techniques.
The paper tackles the problem of minimizing the Conditional Value-at-Risk (CVaR) objective for robust machine learning by developing a stochastic prox-linear (SPL+) method, which better exploits the objective's structure and allows for easier tuning and a wider selection of step sizes compared to the stochastic subgradient method, as supported by experimental results.
We develop a variant of the stochastic prox-linear method for minimizing the Conditional Value-at-Risk (CVaR) objective. CVaR is a risk measure focused on minimizing worst-case performance, defined as the average of the top quantile of the losses. In machine learning, such a risk measure is useful to train more robust models. Although the stochastic subgradient method (SGM) is a natural choice for minimizing the CVaR objective, we show that our stochastic prox-linear (SPL+) algorithm can better exploit the structure of the objective, while still providing a convenient closed form update. Our SPL+ method also adapts to the scaling of the loss function, which allows for easier tuning. We then specialize a general convergence theorem for SPL+ to our setting, and show that it allows for a wider selection of step sizes compared to SGM. We support this theoretical finding experimentally.