Headless Language Models: Learning without Predicting with Contrastive Weight Tying
This addresses the high computational cost and performance limitations in language model pre-training for NLP applications, offering a novel approach with substantial efficiency gains.
The paper tackled the problem of self-supervised pre-training of language models by proposing a method that reconstructs input embeddings contrastively instead of predicting token probabilities, resulting in up to 20 times reduced computational requirements and improved downstream performance with a +1.6 GLUE score increase and +2.7 LAMBADA accuracy improvement.
Self-supervised pre-training of language models usually consists in predicting probability distributions over extensive token vocabularies. In this study, we propose an innovative method that shifts away from probability prediction and instead focuses on reconstructing input embeddings in a contrastive fashion via Constrastive Weight Tying (CWT). We apply this approach to pretrain Headless Language Models in both monolingual and multilingual contexts. Our method offers practical advantages, substantially reducing training computational requirements by up to 20 times, while simultaneously enhancing downstream performance and data efficiency. We observe a significant +1.6 GLUE score increase and a notable +2.7 LAMBADA accuracy improvement compared to classical LMs within similar compute budgets.