BARACK: Partially Supervised Group Robustness With Guarantees
This addresses robustness and fairness issues in machine learning for scenarios where full group annotation is costly, though it is incremental as it builds on existing methods by leveraging partial supervision.
The paper tackles the problem of improving worst-group performance in neural networks when group labels are expensive to obtain, by proposing BARACK, a two-step framework that uses partial group information to achieve better results than baselines without group labels, with empirical gains even when only 1-33% of points have labels.
While neural networks have shown remarkable success on classification tasks in terms of average-case performance, they often fail to perform well on certain groups of the data. Such group information may be expensive to obtain; thus, recent works in robustness and fairness have proposed ways to improve worst-group performance even when group labels are unavailable for the training data. However, these methods generally underperform methods that utilize group information at training time. In this work, we assume access to a small number of group labels alongside a larger dataset without group labels. We propose BARACK, a simple two-step framework to utilize this partial group information to improve worst-group performance: train a model to predict the missing group labels for the training data, and then use these predicted group labels in a robust optimization objective. Theoretically, we provide generalization bounds for our approach in terms of the worst-group performance, which scale with respect to both the total number of training points and the number of training points with group labels. Empirically, our method outperforms the baselines that do not use group information, even when only 1-33% of points have group labels. We provide ablation studies to support the robustness and extensibility of our framework.