Towards More Fine-grained and Reliable NLP Performance Prediction
This work addresses the need for more detailed and trustworthy performance prediction in NLP to reduce experimental overhead, though it is incremental in extending existing methods.
The paper tackles the problem of predicting NLP system performance more precisely by focusing on fine-grained measures like class-specific accuracy and assessing prediction reliability through confidence intervals and calibration, demonstrating feasibility across four NLP tasks.
Performance prediction, the task of estimating a system's performance without performing experiments, allows us to reduce the experimental burden caused by the combinatorial explosion of different datasets, languages, tasks, and models. In this paper, we make two contributions to improving performance prediction for NLP tasks. First, we examine performance predictors not only for holistic measures of accuracy like F1 or BLEU but also fine-grained performance measures such as accuracy over individual classes of examples. Second, we propose methods to understand the reliability of a performance prediction model from two angles: confidence intervals and calibration. We perform an analysis of four types of NLP tasks, and both demonstrate the feasibility of fine-grained performance prediction and the necessity to perform reliability analysis for performance prediction methods in the future. We make our code publicly available: \url{https://github.com/neulab/Reliable-NLPPP}