An Item Response Theory-based R Module for Algorithm Portfolio Analysis
This provides a tool for AI researchers to better analyze algorithm portfolios, though it is incremental as it adapts an existing method from psychometrics to a new domain.
The paper tackles the problem of limited algorithm evaluation in AI by introducing an Item Response Theory-based tool called AIRT-Module, which computes anomalousness, consistency, and difficulty limits for algorithms and visualizes their strengths and weaknesses across test instances to enhance comprehensive assessment.
Experimental evaluation is crucial in AI research, especially for assessing algorithms across diverse tasks. Many studies often evaluate a limited set of algorithms, failing to fully understand their strengths and weaknesses within a comprehensive portfolio. This paper introduces an Item Response Theory (IRT) based analysis tool for algorithm portfolio evaluation called AIRT-Module. Traditionally used in educational psychometrics, IRT models test question difficulty and student ability using responses to test questions. Adapting IRT to algorithm evaluation, the AIRT-Module contains a Shiny web application and the R package airt. AIRT-Module uses algorithm performance measures to compute anomalousness, consistency, and difficulty limits for an algorithm and the difficulty of test instances. The strengths and weaknesses of algorithms are visualised using the difficulty spectrum of the test instances. AIRT-Module offers a detailed understanding of algorithm capabilities across varied test instances, thus enhancing comprehensive AI method assessment. It is available at https://sevvandi.shinyapps.io/AIRT/ .