Dealing with Abbreviations in the Slovenian Biographical Lexicon
This addresses abbreviation-related issues for NLP applications in low-resource languages like Slovenian, but it is incremental as it focuses on a specific domain and dataset.
The paper tackled the problem of abbreviations causing tokenization and out-of-vocabulary errors in NLP systems, especially in low-resource settings, by proposing a new method for identifying and expanding domain-specific abbreviations in Slovenian biographical texts, achieving significant improvements over ad-hoc solutions on a dataset of 51 biographies.
Abbreviations present a significant challenge for NLP systems because they cause tokenization and out-of-vocabulary errors. They can also make the text less readable, especially in reference printed books, where they are extensively used. Abbreviations are especially problematic in low-resource settings, where systems are less robust to begin with. In this paper, we propose a new method for addressing the problems caused by a high density of domain-specific abbreviations in a text. We apply this method to the case of a Slovenian biographical lexicon and evaluate it on a newly developed gold-standard dataset of 51 Slovenian biographies. Our abbreviation identification method performs significantly better than commonly used ad-hoc solutions, especially at identifying unseen abbreviations. We also propose and present the results of a method for expanding the identified abbreviations in context.