Adversarially robust transfer learning
This addresses the problem of computational and data efficiency in adversarial robustness for machine learning practitioners, offering an incremental improvement over existing transfer learning methods.
The paper tackles the challenge of creating adversarially robust neural network models when data is scarce or training is costly, by showing that robust networks contain robust feature extractors that can be transferred to new domains. The result is accurate and robust models achieved with little data and without adversarial training, while maintaining or improving generalization.
Transfer learning, in which a network is trained on one task and re-purposed on another, is often used to produce neural network classifiers when data is scarce or full-scale training is too costly. When the goal is to produce a model that is not only accurate but also adversarially robust, data scarcity and computational limitations become even more cumbersome. We consider robust transfer learning, in which we transfer not only performance but also robustness from a source model to a target domain. We start by observing that robust networks contain robust feature extractors. By training classifiers on top of these feature extractors, we produce new models that inherit the robustness of their parent networks. We then consider the case of fine tuning a network by re-training end-to-end in the target domain. When using lifelong learning strategies, this process preserves the robustness of the source network while achieving high accuracy. By using such strategies, it is possible to produce accurate and robust models with little data, and without the cost of adversarial training. Additionally, we can improve the generalization of adversarially trained models, while maintaining their robustness.