Four Quadrants of Difficulty: A Simple Categorisation and its Limits
This work addresses the challenge of improving curriculum learning for NLP practitioners by revealing limitations in common intuitions about difficulty estimation.
The paper tackled the problem of estimating sample difficulty for curriculum learning in NLP by proposing a four-quadrant categorization of difficulty signals and analyzing their interactions on a natural language understanding dataset, finding that task-agnostic features behave independently and only task-dependent features align with model learning.
Curriculum Learning (CL) aims to improve the outcome of model training by estimating the difficulty of samples and scheduling them accordingly. In NLP, difficulty is commonly approximated using task-agnostic linguistic heuristics or human intuition, implicitly assuming that these signals correlate with what neural models find difficult to learn. We propose a four-quadrant categorisation of difficulty signals -- human vs. model and task-agnostic vs. task-dependent -- and systematically analyse their interactions on a natural language understanding dataset. We find that task-agnostic features behave largely independently and that only task-dependent features align. These findings challenge common CL intuitions and highlight the need for lightweight, task-dependent difficulty estimators that better reflect model learning behaviour.