Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Model
This work addresses the lack of rigorous comparative analysis in the shift from encoder-decoder to decoder-only architectures for large language models, potentially unlocking more efficient and powerful LLMs for AI researchers and practitioners.
The study revisits encoder-decoder large language models (RedLLM) by enhancing them with techniques from decoder-only models, finding that RedLLM shows compelling scaling properties and strong performance, achieving comparable or better results on downstream tasks with substantially better inference efficiency after instruction tuning.
Recent large language model (LLM) research has undergone an architectural shift from encoder-decoder modeling to nowadays the dominant decoder-only modeling. This rapid transition, however, comes without a rigorous comparative analysis especially \textit{from the scaling perspective}, raising concerns that the potential of encoder-decoder models may have been overlooked. To fill this gap, we revisit encoder-decoder LLM (RedLLM), enhancing it with recent recipes from decoder-only LLM (DecLLM). We conduct a comprehensive comparison between RedLLM, pretrained with prefix language modeling (LM), and DecLLM, pretrained with causal LM, at different model scales, ranging from $\sim$150M to $\sim$8B. Using RedPajama V1 (1.6T tokens) for pretraining and FLAN for instruction tuning, our experiments show that RedLLM produces compelling scaling properties and surprisingly strong performance. While DecLLM is overall more compute-optimal during pretraining, RedLLM demonstrates comparable scaling and context length extrapolation capabilities. After instruction tuning, RedLLM achieves comparable and even better results on various downstream tasks while enjoying substantially better inference efficiency. We hope our findings could inspire more efforts on re-examining RedLLM, unlocking its potential for developing powerful and efficient LLMs.