SoK: Analyzing Adversarial Examples: A Framework to Study Adversary Knowledge
This work addresses the problem of inconsistent threat models for researchers in adversarial machine learning, though it is incremental as it builds on existing concepts.
The paper tackles the lack of formalization in studying adversary knowledge for adversarial examples in image classification, resulting in a theoretical framework and systematization that confirms existing beliefs and derives new conclusions, such as transferable attacks being less difficult than previously thought.
Adversarial examples are malicious inputs to machine learning models that trigger a misclassification. This type of attack has been studied for close to a decade, and we find that there is a lack of study and formalization of adversary knowledge when mounting attacks. This has yielded a complex space of attack research with hard-to-compare threat models and attacks. We focus on the image classification domain and provide a theoretical framework to study adversary knowledge inspired by work in order theory. We present an adversarial example game, inspired by cryptographic games, to standardize attacks. We survey recent attacks in the image classification domain and classify their adversary's knowledge in our framework. From this systematization, we compile results that both confirm existing beliefs about adversary knowledge, such as the potency of information about the attacked model as well as allow us to derive new conclusions on the difficulty associated with the white-box and transferable threat models, for example, that transferable attacks might not be as difficult as previously thought.