Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds
This addresses the scalability issue for IRT in NLP by enabling large-scale parameter estimation without human data, though it is incremental as it adapts an existing method to new data sources.
The paper tackled the bottleneck of learning Item Response Theory (IRT) models traditionally requiring human response patterns by proposing to use artificial crowds of deep neural networks to generate response patterns, showing medium to large positive correlations between human and machine parameters for NLP tasks and outperforming baselines in training set filtering.
Incorporating Item Response Theory (IRT) into NLP tasks can provide valuable information about model performance and behavior. Traditionally, IRT models are learned using human response pattern (RP) data, presenting a significant bottleneck for large data sets like those required for training deep neural networks (DNNs). In this work we propose learning IRT models using RPs generated from artificial crowds of DNN models. We demonstrate the effectiveness of learning IRT models using DNN-generated data through quantitative and qualitative analyses for two NLP tasks. Parameters learned from human and machine RPs for natural language inference and sentiment analysis exhibit medium to large positive correlations. We demonstrate a use-case for latent difficulty item parameters, namely training set filtering, and show that using difficulty to sample training data outperforms baseline methods. Finally, we highlight cases where human expectation about item difficulty does not match difficulty as estimated from the machine RPs.