Auditing Fairness by Betting
This work addresses the need for practical and efficient fairness auditing in real-world systems, offering incremental improvements over fixed-sample methods by enabling dynamic monitoring and adaptability to changing data conditions.
The paper tackles the problem of auditing fairness in deployed classification and regression models by introducing sequential, nonparametric methods that allow for continuous monitoring and handle data collected under probabilistic policies and distribution shifts, demonstrating efficacy on three benchmark fairness datasets.
We provide practical, efficient, and nonparametric methods for auditing the fairness of deployed classification and regression models. Whereas previous work relies on a fixed-sample size, our methods are sequential and allow for the continuous monitoring of incoming data, making them highly amenable to tracking the fairness of real-world systems. We also allow the data to be collected by a probabilistic policy as opposed to sampled uniformly from the population. This enables auditing to be conducted on data gathered for another purpose. Moreover, this policy may change over time and different policies may be used on different subpopulations. Finally, our methods can handle distribution shift resulting from either changes to the model or changes in the underlying population. Our approach is based on recent progress in anytime-valid inference and game-theoretic statistics-the "testing by betting" framework in particular. These connections ensure that our methods are interpretable, fast, and easy to implement. We demonstrate the efficacy of our approach on three benchmark fairness datasets.