Predicting Out-of-Distribution Error with the Projection Norm
This addresses the challenge of assessing model reliability in real-world scenarios where data distributions shift, though it is incremental as it builds on prior OOD error prediction methods.
The paper tackles the problem of predicting model performance on out-of-distribution data without ground truth labels by proposing the Projection Norm metric, which outperforms existing methods on image and text classification tasks across various architectures and uniquely achieves non-trivial detection on adversarial examples.
We propose a metric -- Projection Norm -- to predict a model's performance on out-of-distribution (OOD) data without access to ground truth labels. Projection Norm first uses model predictions to pseudo-label test samples and then trains a new model on the pseudo-labels. The more the new model's parameters differ from an in-distribution model, the greater the predicted OOD error. Empirically, our approach outperforms existing methods on both image and text classification tasks and across different network architectures. Theoretically, we connect our approach to a bound on the test error for overparameterized linear models. Furthermore, we find that Projection Norm is the only approach that achieves non-trivial detection performance on adversarial examples. Our code is available at https://github.com/yaodongyu/ProjNorm.