On the Role of Bidirectionality in Language Model Pre-Training
This work addresses the problem of understanding architectural trade-offs in language models for researchers and practitioners, but it is incremental as it builds on prior approaches like GPT and BERT.
The authors studied the role of bidirectionality in language model pre-training, finding that its optimal configuration depends on the application, with bidirectional attention beneficial for fine-tuning and text infilling but harmful for next token prediction and zero-shot priming, and these differences remained consistent in models up to 6.7B parameters.
Prior work on language model pre-training has explored different architectures and learning objectives, but differences in data, hyperparameters and evaluation make a principled comparison difficult. In this work, we focus on bidirectionality as a key factor that differentiates existing approaches, and present a comprehensive study of its role in next token prediction, text infilling, zero-shot priming and fine-tuning. We propose a new framework that generalizes prior approaches, including fully unidirectional models like GPT, fully bidirectional models like BERT, and hybrid models like CM3 and prefix LM. Our framework distinguishes between two notions of bidirectionality (bidirectional context and bidirectional attention) and allows us to control each of them separately. We find that the optimal configuration is largely application-dependent (e.g., bidirectional attention is beneficial for fine-tuning and infilling, but harmful for next token prediction and zero-shot priming). We train models with up to 6.7B parameters, and find differences to remain consistent at scale. While prior work on scaling has focused on left-to-right autoregressive models, our results suggest that this approach comes with some trade-offs, and it might be worthwhile to develop very large bidirectional models.