Robust Feature Learning for Multi-Index Models in High Dimensions
This addresses the challenge of adversarial robustness in machine learning for high-dimensional data, offering a theoretical guarantee that robust learning is as efficient as standard learning, which is incremental but provides new insights for secure AI applications.
The paper tackles the problem of adversarially robust feature learning for multi-index models in high dimensions, proving that hidden directions provide a Bayes optimal low-dimensional projection for robustness against ℓ₂-bounded perturbations, and shows that robust learning requires no additional samples beyond standard learning, independent of dimensionality.
Recently, there have been numerous studies on feature learning with neural networks, specifically on learning single- and multi-index models where the target is a function of a low-dimensional projection of the input. Prior works have shown that in high dimensions, the majority of the compute and data resources are spent on recovering the low-dimensional projection; once this subspace is recovered, the remainder of the target can be learned independently of the ambient dimension. However, implications of feature learning in adversarial settings remain unexplored. In this work, we take the first steps towards understanding adversarially robust feature learning with neural networks. Specifically, we prove that the hidden directions of a multi-index model offer a Bayes optimal low-dimensional projection for robustness against $\ell_2$-bounded adversarial perturbations under the squared loss, assuming that the multi-index coordinates are statistically independent from the rest of the coordinates. Therefore, robust learning can be achieved by first performing standard feature learning, then robustly tuning a linear readout layer on top of the standard representations. In particular, we show that adversarially robust learning is just as easy as standard learning. Specifically, the additional number of samples needed to robustly learn multi-index models when compared to standard learning does not depend on dimensionality.