Robust Assessment of Real-World Adversarial Examples
This addresses the need for more rigorous assessment methods for adversarial examples in machine learning security, though it appears incremental as it builds on existing evaluation practices.
The paper tackles the problem of evaluating adversarial examples in real-world settings by proposing a testing regimen and score that accounts for scene changes and baseline performance, showing that small environmental perturbations cause large adversarial performance differences.
We explore rigorous, systematic, and controlled experimental evaluation of adversarial examples in the real world and propose a testing regimen for evaluation of real world adversarial objects. We show that for small scene/ environmental perturbations, large adversarial performance differences exist. Current state of adversarial reporting exists largely as a frequency count over a dynamic collections of scenes. Our work underscores the need for either a more complete report or a score that incorporates scene changes and baseline performance for models and environments tested by adversarial developers. We put forth a score that attempts to address the above issues in a straight-forward exemplar application for multiple generated adversary examples. We contribute the following: 1. a testbed for adversarial assessment, 2. a score for adversarial examples, and 3. a collection of additional evaluations on testbed data.