A law of adversarial risk, interpolation, and label noise
This work addresses the problem of adversarial robustness for machine learning practitioners by revealing a fundamental trade-off between interpolation and security, though it is incremental in building on prior label noise studies.
The paper demonstrates that interpolating label noise in supervised learning leads to adversarial vulnerability, proving a theorem linking label noise to adversarial risk for any data distribution, with results showing uniform label noise induces nearly as large an adversarial risk as worst-case poisoning.
In supervised learning, it has been shown that label noise in the data can be interpolated without penalties on test accuracy. We show that interpolating label noise induces adversarial vulnerability, and prove the first theorem showing the relationship between label noise and adversarial risk for any data distribution. Our results are almost tight if we do not make any assumptions on the inductive bias of the learning algorithm. We then investigate how different components of this problem affect this result, including properties of the distribution. We also discuss non-uniform label noise distributions; and prove a new theorem showing uniform label noise induces nearly as large an adversarial risk as the worst poisoning with the same noise rate. Then, we provide theoretical and empirical evidence that uniform label noise is more harmful than typical real-world label noise. Finally, we show how inductive biases amplify the effect of label noise and argue the need for future work in this direction.