Using Synthetic Corruptions to Measure Robustness to Natural Distribution Shifts
This work addresses the challenge of accurately measuring robustness for machine learning practitioners, though it is incremental as it refines existing benchmark construction methods.
The paper tackles the problem that synthetic corruption benchmarks often fail to predict neural network robustness to real-world distribution shifts, and proposes a methodology to build more predictive synthetic corruption benchmarks, resulting in new selections that outperform existing benchmarks.
Synthetic corruptions gathered into a benchmark are frequently used to measure neural network robustness to distribution shifts. However, robustness to synthetic corruption benchmarks is not always predictive of robustness to distribution shifts encountered in real-world applications. In this paper, we propose a methodology to build synthetic corruption benchmarks that make robustness estimations more correlated with robustness to real-world distribution shifts. Using the overlapping criterion, we split synthetic corruptions into categories that help to better understand neural network robustness. Based on these categories, we identify three relevant parameters to take into account when constructing a corruption benchmark that are the (1) number of represented categories, (2) their relative balance in terms of size and, (3) the size of the considered benchmark. In doing so, we build new synthetic corruption selections that are more predictive of robustness to natural corruptions than existing synthetic corruption benchmarks.