Standing on the shoulders of giants
This provides a complementary evaluation layer for machine learning practitioners to better distinguish between models with similar performance, though it is incremental as it builds on existing IRT concepts.
The paper tackles the limitation of classic evaluation metrics like precision and F1 by integrating psychometric metrics such as Item Response Theory (IRT) to assess model behavior at the instance level, finding that IRT scores differ from 66% of classical metrics with 97% confidence.
Although fundamental to the advancement of Machine Learning, the classic evaluation metrics extracted from the confusion matrix, such as precision and F1, are limited. Such metrics only offer a quantitative view of the models' performance, without considering the complexity of the data or the quality of the hit. To overcome these limitations, recent research has introduced the use of psychometric metrics such as Item Response Theory (IRT), which allows an assessment at the level of latent characteristics of instances. This work investigates how IRT concepts can enrich a confusion matrix in order to identify which model is the most appropriate among options with similar performance. In the study carried out, IRT does not replace, but complements classical metrics by offering a new layer of evaluation and observation of the fine behavior of models in specific instances. It was also observed that there is 97% confidence that the score from the IRT has different contributions from 66% of the classical metrics analyzed.