Cup Curriculum: Curriculum Learning on Model Capacity
This addresses a gap in curriculum learning for NLP practitioners, though it is incremental as it adapts existing pruning techniques.
The paper tackles the lack of curriculum learning applied to model capacity in NLP by proposing the cup curriculum, which reduces and then reintroduces model weights, resulting in improved performance over early stopping and high resilience to overfitting.
Curriculum learning (CL) aims to increase the performance of a learner on a given task by applying a specialized learning strategy. This strategy focuses on either the dataset, the task, or the model. There is little to no work analysing the possibilities to apply CL on the model capacity in natural language processing. To close this gap, we propose the cup curriculum. In a first phase of training we use a variation of iterative magnitude pruning to reduce model capacity. These weights are reintroduced in a second phase, resulting in the model capacity to show a cup-shaped curve over the training iterations. We empirically evaluate different strategies of the cup curriculum and show that it outperforms early stopping reliably while exhibiting a high resilience to overfitting.