MiniLingua: A Small Open-Source LLM for European Languages
This addresses the problem of high computational cost and English-centric bias for users needing efficient, multilingual AI, though it is incremental as it builds on existing small model approaches.
The paper tackles the limitations of large language models by introducing MiniLingua, a small open-source multilingual LLM with one billion parameters trained for 13 European languages, which outperforms EuroLLM on tasks like summarization and question answering and remains competitive with state-of-the-art models in open-ended generation.
Large language models are powerful but often limited by high computational cost, privacy concerns, and English-centric training. Recent progress demonstrates that small, efficient models with around one billion parameters can deliver strong results and enable on-device use. This paper introduces MiniLingua, a multilingual open-source LLM of one billion parameters trained from scratch for 13 European languages, designed to balance coverage and instruction-following capabilities. Based on evaluation results, the instruction-tuned version of MiniLingua outperforms EuroLLM, a model with a similar training approach but a larger training budget, on summarization, classification and both open- and closed-book question answering. Moreover, it remains competitive with more advanced state-of-the-art models on open-ended generation tasks. We release model weights, tokenizer and source code used for data processing and model training.