Training Well-Generalizing Classifiers for Fairness Metrics and Other Data-Dependent Constraints
This addresses generalization issues for practitioners applying constrained optimization to real-world problems like fairness, but it is incremental as it builds on existing two-player constrained optimization methods.
The paper tackles the problem of poor generalization when training classifiers with data-dependent constraints (like fairness metrics), showing that a two-player game approach using separate training and validation datasets significantly improves how well constraints are satisfied at evaluation time.
Classifiers can be trained with data-dependent constraints to satisfy fairness goals, reduce churn, achieve a targeted false positive rate, or other policy goals. We study the generalization performance for such constrained optimization problems, in terms of how well the constraints are satisfied at evaluation time, given that they are satisfied at training time. To improve generalization performance, we frame the problem as a two-player game where one player optimizes the model parameters on a training dataset, and the other player enforces the constraints on an independent validation dataset. We build on recent work in two-player constrained optimization to show that if one uses this two-dataset approach, then constraint generalization can be significantly improved. As we illustrate experimentally, this approach works not only in theory, but also in practice.