The advantages of context specific language models: the case of the Erasmian Language Model
This work addresses resource-constrained and privacy-sensitive use cases for institutions and organizations, offering a viable alternative to large-scale models, though it is incremental in applying existing methods to a new context.
The paper tackles the high computational and privacy costs of large language models by introducing the Erasmian Language Model, a 900-million-parameter context-specific model for Erasmus University Rotterdam, which performs adequately in classroom essay writing and achieves superior performance in its specific subjects.
The current trend to improve language model performance seems to be based on scaling up with the number of parameters (e.g. the state of the art GPT4 model has approximately 1.7 trillion parameters) or the amount of training data fed into the model. However this comes at significant costs in terms of computational resources and energy costs that compromise the sustainability of AI solutions, as well as risk relating to privacy and misuse. In this paper we present the Erasmian Language Model (ELM) a small context specific, 900 million parameter model, pre-trained and fine-tuned by and for Erasmus University Rotterdam. We show how the model performs adequately in a classroom context for essay writing, and how it achieves superior performance in subjects that are part of its context. This has implications for a wide range of institutions and organizations, showing that context specific language models may be a viable alternative for resource constrained, privacy sensitive use cases.