Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
This work addresses the need for reproducible and comparable adversarial benchmarks in the machine learning community, though it is incremental as it updates an existing library.
The authors tackled the problem of non-comparable benchmarks in adversarial machine learning by releasing CleverHans v2.1.0, a library providing standardized implementations of adversarial attacks and defenses to enable consistent evaluation of model robustness.
CleverHans is a software library that provides standardized reference implementations of adversarial example construction techniques and adversarial training. The library may be used to develop more robust machine learning models and to provide standardized benchmarks of models' performance in the adversarial setting. Benchmarks constructed without a standardized implementation of adversarial example construction are not comparable to each other, because a good result may indicate a robust model or it may merely indicate a weak implementation of the adversarial example construction procedure. This technical report is structured as follows. Section 1 provides an overview of adversarial examples in machine learning and of the CleverHans software. Section 2 presents the core functionalities of the library: namely the attacks based on adversarial examples and defenses to improve the robustness of machine learning models to these attacks. Section 3 describes how to report benchmark results using the library. Section 4 describes the versioning system.