NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Framework
This addresses the problem of expensive training for NLP practitioners by offering a more efficient alternative to standard pretrained models, though it appears incremental as it builds on existing retrieval and joint optimization ideas.
The paper tackles the high computational cost of pretrained language models by proposing TLM, a framework that avoids large-scale pretraining and instead retrieves a small subset of a general corpus using task data as queries, then jointly optimizes task and language modeling objectives from scratch. It achieves results comparable to or better than RoBERTa-Large on eight classification datasets while reducing training FLOPs by two orders of magnitude.
Pretrained language models have become the standard approach for many NLP tasks due to strong performance, but they are very expensive to train. We propose a simple and efficient learning framework, TLM, that does not rely on large-scale pretraining. Given some labeled task data and a large general corpus, TLM uses task data as queries to retrieve a tiny subset of the general corpus and jointly optimizes the task objective and the language modeling objective from scratch. On eight classification datasets in four domains, TLM achieves results better than or similar to pretrained language models (e.g., RoBERTa-Large) while reducing the training FLOPs by two orders of magnitude. With high accuracy and efficiency, we hope TLM will contribute to democratizing NLP and expediting its development.