Beyond Random Sampling: Instance Quality-Based Data Partitioning via Item Response Theory
This work addresses the need for better data partitioning in ML validation, offering a domain-specific method that is incremental but provides concrete performance insights.
The study tackled the problem of robust model validation by proposing Item Response Theory (IRT) parameters to guide data partitioning, revealing that IRT-informed partitions expose instance heterogeneity and improve understanding of bias-variance tradeoffs, with accuracy varying from below 50% to over 70% based on guessing parameters.
Robust validation of Machine Learning (ML) models is essential, but traditional data partitioning approaches often ignore the intrinsic quality of each instance. This study proposes the use of Item Response Theory (IRT) parameters to characterize and guide the partitioning of datasets in the model validation stage. The impact of IRT-informed partitioning strategies on the performance of several ML models in four tabular datasets was evaluated. The results obtained demonstrate that IRT reveals an inherent heterogeneity of the instances and highlights the existence of informative subgroups of instances within the same dataset. Based on IRT, balanced partitions were created that consistently help to better understand the tradeoff between bias and variance of the models. In addition, the guessing parameter proved to be a determining factor: training with high-guessing instances can significantly impair model performance and resulted in cases with accuracy below 50%, while other partitions reached more than 70% in the same dataset.