Amulet: a Python Library for Assessing Interactions Among ML Defenses and Risks
This addresses the need for researchers and practitioners to systematically evaluate how defenses against ML risks interact, which is incremental as it builds on existing defenses and risks.
The authors tackled the problem of unintended interactions among machine learning defenses and risks by introducing Amulet, a Python library that provides a unified foundation for evaluating both intended and unintended interactions, enabling systematic assessment across multiple risks.
Machine learning (ML) models are susceptible to various risks to security, privacy, and fairness. Most defenses are designed to protect against each risk individually (intended interactions) but can inadvertently affect susceptibility to other unrelated risks (unintended interactions). We introduce Amulet, the first Python library for evaluating both intended and unintended interactions among ML defenses and risks. Amulet is comprehensive by including representative attacks, defenses, and metrics; extensible to new modules due to its modular design; consistent with a user-friendly API template for inputs and outputs; and applicable for evaluating novel interactions. By satisfying all four properties, Amulet offers a unified foundation for studying how defenses interact, enabling the first systematic evaluation of unintended interactions across multiple risks.