Albanian Language Identification in Text Documents
This work addresses language identification for Albanian, an incremental improvement focusing on handling missing characters in a specific domain.
The study tackled the problem of accurately identifying Albanian language in text documents, particularly when documents lack specific Albanian alphabet letters like 'Ë' and 'Ç', by constructing a custom training corpus that achieved over 99% accuracy.
In this work we investigate the accuracy of standard and state-of-the-art language identification methods in identifying Albanian in written text documents. A dataset consisting of news articles written in Albanian has been constructed for this purpose. We noticed a considerable decrease of accuracy when using test documents that miss the Albanian alphabet letters " Ë " and " Ç " and created a custom training corpus that solved this problem by achieving an accuracy of more than 99%. Based on our experiments, the most performing language identification methods for Albanian use a naïve Bayes classifier and n-gram based classification features.