Source-Free Domain-Invariant Performance Prediction
This addresses a critical challenge for machine learning practitioners in realistic deployment scenarios where only trained models are accessible, representing a notable advance over prior methods.
The paper tackles the problem of accurately predicting model performance when source data is unavailable and target domains differ, proposing a source-free uncertainty-based method that significantly outperforms existing state-of-the-art approaches in benchmark object recognition datasets.
Accurately estimating model performance poses a significant challenge, particularly in scenarios where the source and target domains follow different data distributions. Most existing performance prediction methods heavily rely on the source data in their estimation process, limiting their applicability in a more realistic setting where only the trained model is accessible. The few methods that do not require source data exhibit considerably inferior performance. In this work, we propose a source-free approach centred on uncertainty-based estimation, using a generative model for calibration in the absence of source data. We establish connections between our approach for unsupervised calibration and temperature scaling. We then employ a gradient-based strategy to evaluate the correctness of the calibrated predictions. Our experiments on benchmark object recognition datasets reveal that existing source-based methods fall short with limited source sample availability. Furthermore, our approach significantly outperforms the current state-of-the-art source-free and source-based methods, affirming its effectiveness in domain-invariant performance estimation.