Dynamic Data Selection for Curriculum Learning via Ability Estimation
This work addresses curriculum learning for NLP practitioners, offering incremental improvements over existing heuristic approaches.
The authors tackled the problem of curriculum learning by replacing heuristic difficulty estimation with learned parameters and a dynamic data selection strategy based on model ability, resulting in improved performance over heuristic-based methods on GLUE classification tasks.
Curriculum learning methods typically rely on heuristics to estimate the difficulty of training examples or the ability of the model. In this work, we propose replacing difficulty heuristics with learned difficulty parameters. We also propose Dynamic Data selection for Curriculum Learning via Ability Estimation (DDaCLAE), a strategy that probes model ability at each training epoch to select the best training examples at that point. We show that models using learned difficulty and/or ability outperform heuristic-based curriculum learning models on the GLUE classification tasks.