Cost-Sensitive Self-Training for Optimizing Non-Decomposable Metrics
This addresses the need for practical machine learning systems to handle complex goals beyond accuracy, such as fairness or robustness, in semi-supervised settings, representing a novel method for a known bottleneck.
The paper tackles the problem of optimizing non-decomposable metrics like maximizing minimum recall across classes in self-training for semi-supervised learning, introducing the Cost-Sensitive Self-Training (CSST) framework that improves over state-of-the-art methods in most cases across datasets and objectives.
Self-training based semi-supervised learning algorithms have enabled the learning of highly accurate deep neural networks, using only a fraction of labeled data. However, the majority of work on self-training has focused on the objective of improving accuracy, whereas practical machine learning systems can have complex goals (e.g. maximizing the minimum of recall across classes, etc.) that are non-decomposable in nature. In this work, we introduce the Cost-Sensitive Self-Training (CSST) framework which generalizes the self-training-based methods for optimizing non-decomposable metrics. We prove that our framework can better optimize the desired non-decomposable metric utilizing unlabeled data, under similar data distribution assumptions made for the analysis of self-training. Using the proposed CSST framework, we obtain practical self-training methods (for both vision and NLP tasks) for optimizing different non-decomposable metrics using deep neural networks. Our results demonstrate that CSST achieves an improvement over the state-of-the-art in majority of the cases across datasets and objectives.