Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing
This work provides a principled solution for improving model performance through retraining, which is incremental but addresses a fundamental open question in binary classification tasks.
The paper tackles the problem of optimally combining a model's predictions with noisy labels during retraining for binary classification, deriving a Bayes optimal aggregator function that minimizes prediction error and demonstrating its superiority in high label noise regimes.
Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance. While prior works have demonstrated the benefits of specific heuristic retraining schemes, the question of how to optimally combine the model's predictions and the provided labels remains largely open. This paper addresses this fundamental question for binary classification tasks. We develop a principled framework based on approximate message passing (AMP) to analyze iterative retraining procedures for two ground truth settings: Gaussian mixture model (GMM) and generalized linear model (GLM). Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels, which when used to retrain the same model, minimizes its prediction error. We also quantify the performance of this optimal retraining strategy over multiple rounds. We complement our theoretical results by proposing a practically usable version of the theoretically-optimal aggregator function for linear probing with the cross-entropy loss, and demonstrate its superiority over baseline methods in the high label noise regime.