Bootstrapping Embeddings for Low Resource Languages
This provides a scalable solution for producing performant embedding models in hundreds of low-resource languages, addressing a critical gap in NLP.
The paper tackles the problem of creating effective embedding models for low-resource languages lacking supervised finetuning data by testing synthetic triplet generation strategies using large language models, finding that adapter composition and XL-LoRA yield strong performance gains across tasks and languages.
Embedding models are crucial to modern NLP. However, the creation of the most effective models relies on carefully constructed supervised finetuning data. For high resource languages, such as English, such datasets are readily available. However, for hundreds of other languages, they are simply non-existent. We investigate whether the advent of large language models can help to bridge this gap. We test three different strategies for generating synthetic triplet data used to optimise embedding models. These include in-context learning as well as two novel approaches, leveraging adapter composition and cross lingual finetuning of the LLM generator (XL-LoRA) respectively. We find that while in-context learning still falls short of strong non-synthetic baselines, adapter composition and XL-LoRA yield strong performance gains across a wide array of tasks and languages, offering a clear, scalable pathway to producing performant embedding models for a wide variety of languages.