Efficient NLP Model Finetuning via Multistage Data Filtering
This addresses efficiency issues for practitioners finetuning large NLP models, though it is incremental as it builds on existing finetuning techniques.
The paper tackles the problem of inefficient NLP model finetuning by proposing a multistage data filtering method that skips training on redundant examples, achieving up to 5.3x reduction in training examples and 6.8x reduction in training time with minor accuracy loss.
As model finetuning is central to the modern NLP, we set to maximize its efficiency. Motivated by redundancy in training examples and the sheer sizes of pretrained models, we exploit a key opportunity: training only on important data. To this end, we set to filter training examples in a streaming fashion, in tandem with training the target model. Our key techniques are two: (1) automatically determine a training loss threshold for skipping backward training passes; (2) run a meta predictor for further skipping forward training passes. We integrate the above techniques in a holistic, three-stage training process. On a diverse set of benchmarks, our method reduces the required training examples by up to 5.3$\times$ and training time by up to 6.8$\times$, while only seeing minor accuracy degradation. Our method is effective even when training one epoch, where each training example is encountered only once. It is simple to implement and is compatible with the existing finetuning techniques. Code is available at: https://github.com/xo28/efficient- NLP-multistage-training