OLMo: Accelerating the Science of Language Models
This addresses the need for accessible, transparent models for the research community to study biases and risks, though it is incremental in providing open resources.
The authors tackled the problem of closed, proprietary language models by building OLMo, a competitive open language model, and released it with open training data, training, and evaluation code to enable scientific study.
Language models (LMs) have become ubiquitous in both NLP research and in commercial product offerings. As their commercial importance has surged, the most powerful models have become closed off, gated behind proprietary interfaces, with important details of their training data, architectures, and development undisclosed. Given the importance of these details in scientifically studying these models, including their biases and potential risks, we believe it is essential for the research community to have access to powerful, truly open LMs. To this end, we have built OLMo, a competitive, truly Open Language Model, to enable the scientific study of language models. Unlike most prior efforts that have only released model weights and inference code, we release OLMo alongside open training data and training and evaluation code. We hope this release will empower the open research community and inspire a new wave of innovation.