Architecture Selection via the Trade-off Between Accuracy and Robustness
This work addresses the problem of selecting optimal architectures for robust machine learning models, though it is incremental as it builds on existing adversarial training and regularization concepts.
The paper tackles the trade-off between accuracy and robustness in supervised learning by proposing a framework to characterize it for different architectures, showing how adversarial training regularizes parameters in linear models and demonstrating trade-off curves for neural networks based on factors like layers and neurons.
We provide a general framework for characterizing the trade-off between accuracy and robustness in supervised learning. We propose a method and define quantities to characterize the trade-off between accuracy and robustness for a given architecture, and provide theoretical insight into the trade-off. Specifically we introduce a simple trade-off curve, define and study an influence function that captures the sensitivity, under adversarial attack, of the optima of a given loss function. We further show how adversarial training regularizes the parameters in an over-parameterized linear model, recovering the LASSO and ridge regression as special cases, which also allows us to theoretically analyze the behavior of the trade-off curve. In experiments, we demonstrate the corresponding trade-off curves of neural networks and how they vary with respect to factors such as number of layers, neurons, and across different network structures. Such information provides a useful guideline to architecture selection.