GliLem: Leveraging GliNER for Contextualized Lemmatization in Estonian
This work addresses lemmatization accuracy for Estonian, a low-resource language, but is incremental as it builds on existing tools like Vabamorf and GliNER.
The authors tackled the problem of lemmatization in Estonian by developing GliLem, a hybrid system that enhances the rule-based Vabamorf analyzer with a GliNER-based disambiguation module, resulting in a 10% improvement in lemmatization accuracy over the original module and showing small but consistent gains in information retrieval recall.
We present GliLem -- a novel hybrid lemmatization system for Estonian that enhances the highly accurate rule-based morphological analyzer Vabamorf with an external disambiguation module based on GliNER -- an open vocabulary NER model that is able to match text spans with text labels in natural language. We leverage the flexibility of a pre-trained GliNER model to improve the lemmatization accuracy of Vabamorf by 10% compared to its original disambiguation module and achieve an improvement over the token classification-based baseline. To measure the impact of improvements in lemmatization accuracy on the information retrieval downstream task, we first created an information retrieval dataset for Estonian by automatically translating the DBpedia-Entity dataset from English. We benchmark several token normalization approaches, including lemmatization, on the created dataset using the BM25 algorithm. We observe a substantial improvement in IR metrics when using lemmatization over simplistic stemming. The benefits of improving lemma disambiguation accuracy manifest in small but consistent improvement in the IR recall measure, especially in the setting of high k.