Robust Sampling in Deep Learning
This work addresses regularization for deep learning practitioners, but it appears incremental as it builds on existing robust optimization techniques.
The paper tackles overfitting in deep learning by introducing a regularization method based on distributional robust optimization, which adjusts sample contributions to tighten empirical risk bounds, resulting in faster convergence or increased accuracy in certain scenarios.
Deep learning requires regularization mechanisms to reduce overfitting and improve generalization. We address this problem by a new regularization method based on distributional robust optimization. The key idea is to modify the contribution from each sample for tightening the empirical risk bound. During the stochastic training, the selection of samples is done according to their accuracy in such a way that the worst performed samples are the ones that contribute the most in the optimization. We study different scenarios and show the ones where it can make the convergence faster or increase the accuracy.