Maximizing Efficiency of Language Model Pre-training for Learning Representation
This work addresses efficiency issues in language model pre-training for researchers and practitioners, but it is incremental as it builds on existing methods like ELECTRA.
The paper tackles the compute inefficiency of pre-trained language models like ELECTRA by proposing an adaptive early exit strategy to leverage earlier layer representations, reducing processing in subsequent layers, but the initial approach failed to maintain model accuracy while showing promising compute efficiency.
Pre-trained language models in the past years have shown exponential growth in model parameters and compute time. ELECTRA is a novel approach for improving the compute efficiency of pre-trained language models (e.g. BERT) based on masked language modeling (MLM) by addressing the sample inefficiency problem with the replaced token detection (RTD) task. Our work proposes adaptive early exit strategy to maximize the efficiency of the pre-training process by relieving the model's subsequent layers of the need to process latent features by leveraging earlier layer representations. Moreover, we evaluate an initial approach to the problem that has not succeeded in maintaining the accuracy of the model while showing a promising compute efficiency by thoroughly investigating the necessity of the generator module of ELECTRA.