Low-resource Bilingual Dialect Lexicon Induction with Large Language Models
This work addresses low-resource bilingual lexicon induction for German dialects, which is incremental as it applies existing methods to new data.
The paper tackled bilingual lexicon induction for German and its dialects Bavarian and Alemannic, addressing challenges like resource scarcity and orthographic variation, and released evaluation datasets of 1,500 sentence pairs and 1,000 word pairs for assessment.
Bilingual word lexicons are crucial tools for multilingual natural language understanding and machine translation tasks, as they facilitate the mapping of words in one language to their synonyms in another language. To achieve this, numerous papers have explored bilingual lexicon induction (BLI) in high-resource scenarios, using a typical pipeline consisting of two unsupervised steps: bitext mining and word alignment, both of which rely on pre-trained large language models~(LLMs). In this paper, we present an analysis of the BLI pipeline for German and two of its dialects, Bavarian and Alemannic. This setup poses several unique challenges, including the scarcity of resources, the relatedness of the languages, and the lack of standardization in the orthography of dialects. To evaluate the BLI outputs, we analyze them with respect to word frequency and pairwise edit distance. Additionally, we release two evaluation datasets comprising 1,500 bilingual sentence pairs and 1,000 bilingual word pairs. They were manually judged for their semantic similarity for each Bavarian-German and Alemannic-German language pair.