ILDAE: Instance-Level Difficulty Analysis of Evaluation Data
This work addresses the need for more efficient and reliable evaluation in NLP, offering practical tools for researchers and practitioners, though it is incremental in applying difficulty analysis to existing datasets.
The paper tackles the problem of leveraging instance-level difficulty in NLP evaluation data, demonstrating that using just 5% of instances selected via their method achieves a 0.93 Kendall correlation with full-dataset evaluation and improves correlation with out-of-domain performance by 5.2%.
Knowledge of questions' difficulty level helps a teacher in several ways, such as estimating students' potential quickly by asking carefully selected questions and improving quality of examination by modifying trivial and hard questions. Can we extract such benefits of instance difficulty in NLP? To this end, we conduct Instance-Level Difficulty Analysis of Evaluation data (ILDAE) in a large-scale setup of 23 datasets and demonstrate its five novel applications: 1) conducting efficient-yet-accurate evaluations with fewer instances saving computational cost and time, 2) improving quality of existing evaluation datasets by repairing erroneous and trivial instances, 3) selecting the best model based on application requirements, 4) analyzing dataset characteristics for guiding future data creation, 5) estimating Out-of-Domain performance reliably. Comprehensive experiments for these applications result in several interesting findings, such as evaluation using just 5% instances (selected via ILDAE) achieves as high as 0.93 Kendall correlation with evaluation using complete dataset and computing weighted accuracy using difficulty scores leads to 5.2% higher correlation with Out-of-Domain performance. We release the difficulty scores and hope our analyses and findings will bring more attention to this important yet understudied field of leveraging instance difficulty in evaluations.