OLMoE: Open Mixture-of-Experts Language Models
This provides an efficient, open-source solution for researchers and practitioners needing high-performance language models with reduced computational costs.
The authors tackled the problem of scaling language models efficiently by introducing OLMoE, an open Mixture-of-Experts model that uses 1 billion active parameters per token to achieve performance surpassing larger models like Llama2-13B-Chat and DeepSeekMoE-16B, as demonstrated through pretraining on 5 trillion tokens.
We introduce OLMoE, a fully open, state-of-the-art language model leveraging sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but uses only 1B per input token. We pretrain it on 5 trillion tokens and further adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available models with similar active parameters, even surpassing larger ones like Llama2-13B-Chat and DeepSeekMoE-16B. We present various experiments on MoE training, analyze routing in our model showing high specialization, and open-source all aspects of our work: model weights, training data, code, and logs.