Can We Reliably Rank Model Performance across Domains without Labeled Data?
This addresses the challenge of cross-domain model evaluation for NLP practitioners, providing guidance on when performance estimates can be trusted, though it is incremental as it builds on prior work on error prediction.
The paper tackled the problem of reliably ranking model performance across domains without labeled data, finding that large language model-based error predictors produce stronger rank correlations with true accuracy than baselines, with experiments showing up to 0.85 correlation on 15 domains.
Estimating model performance without labels is an important goal for understanding how NLP models generalize. While prior work has proposed measures based on dataset similarity or predicted correctness, it remains unclear when these estimates produce reliable performance rankings across domains. In this paper, we analyze the factors that affect ranking reliability using a two-step evaluation setup with four base classifiers and several large language models as error predictors. Experiments on the GeoOLID and Amazon Reviews datasets, spanning 15 domains, show that large language model-based error predictors produce stronger and more consistent rank correlations with true accuracy than drift-based or zero-shot baselines. Our analysis reveals two key findings: ranking is more reliable when performance differences across domains are larger, and when the error model's predictions align with the base model's true failure patterns. These results clarify when performance estimation methods can be trusted and provide guidance for their use in cross-domain model evaluation.