PaPaformer: Language Model from Pre-trained Parallel Paths
This addresses the need for faster and more efficient training of language models, particularly for resource-constrained settings, though it is incremental as it builds on existing transformer architectures.
The paper tackles the problem of high computational cost and long training times for language models by introducing PaPaformer, a decoder-only transformer variant that trains lower-dimensional parallel paths separately and combines them, achieving reduced parameters and training time with improved performance.
The training of modern large-language models requires an increasingly amount of computation power and time. Even smaller variants, such as small-language models (SLMs), take several days to train in the best-case scenarios, often requiring multiple GPUs. This paper explores methods to train and evaluate decoder-only transformer-based language models in hours instead of days/weeks. We introduces \textit{PaPaformer}, a decoder-only transformer architecture variant, whose lower-dimensional parallel paths are combined into larger model. The paper shows that these lower-dimensional paths can be trained individually with different types of training data and then combined into one larger model. This method gives the option to reduce the total number of model parameters and the training time with increasing performance. Moreover, the use of parallel path structure opens interesting possibilities to customize paths to accommodate specific task requirements.