Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian
This work addresses the challenge of language-specific text generation for Russian users, representing an incremental improvement over previous adapter-based methods.
The authors tackled the problem of poor text generation quality and computational inefficiency for non-English languages in large language models by developing a pipeline to adapt English-oriented pre-trained models, resulting in Vikhr, a series of bilingual open-source instruction-tuned LLMs for Russian that show enhanced performance and efficiency.
There has been a surge in the development of various Large Language Models (LLMs). However, text generation for languages other than English often faces significant challenges, including poor generation quality and reduced computational performance due to the disproportionate representation of tokens in the model's vocabulary. In this work, we address these issues by developing a pipeline for the adaptation of English-oriented pre-trained models to other languages and constructing efficient bilingual LLMs. Using this pipeline, we construct Vikhr, a series of bilingual open-source instruction-following LLMs designed specifically for the Russian language. ``Vikhr'' refers to the name of the Mistral LLM series and means a ``strong gust of wind.'' Unlike previous Russian-language models that typically rely on LoRA adapters on top of English-oriented models, sacrificing performance for lower training costs, Vikhr features an adapted tokenizer vocabulary and undergoes the continued pre-training and instruction tuning of all weights. This not only enhances the model's performance but also significantly improves its computational and contextual efficiency. We also expanded the instruction datasets and corpora for continued pre-training. The model weights, instruction sets, and code are publicly available.