Learning Against Distributional Uncertainty: On the Trade-off Between Robustness and Specificity
It addresses the challenge of trustworthy machine learning for practitioners dealing with uncertain data distributions, though it appears incremental as it builds on existing approaches.
This paper tackles the problem of distributional uncertainty in machine learning by proposing a new framework that unifies Bayesian, distributionally robust optimization, and regularization methods, addressing issues like prior specification, conservatism, and bias, and demonstrates its superiority in experiments on real-world tasks.
Trustworthy machine learning aims at combating distributional uncertainties in training data distributions compared to population distributions. Typical treatment frameworks include the Bayesian approach, (min-max) distributionally robust optimization (DRO), and regularization. However, three issues have to be raised: 1) the prior distribution in the Bayesian method and the regularizer in the regularization method are difficult to specify; 2) the DRO method tends to be overly conservative; 3) all the three methods are biased estimators of the true optimal cost. This paper studies a new framework that unifies the three approaches and addresses the three challenges above. The asymptotic properties (e.g., consistencies and asymptotic normalities), non-asymptotic properties (e.g., generalization bounds and unbiasedness), and solution methods of the proposed model are studied. The new model reveals the trade-off between the robustness to the unseen data and the specificity to the training data. Experiments on various real-world tasks validate the superiority of the proposed learning framework.