Evaluating Named Entity Recognition Models for Russian Cultural News Texts: From BERT to LLM
This addresses the problem of NER in morphologically rich languages like Russian for researchers and practitioners in the cultural heritage domain, but it is incremental as it applies existing methods to a new dataset.
This paper tackled Named Entity Recognition for person names in Russian cultural news texts, finding that GPT-4o achieved an F1 score of 0.93 with specific prompting, and GPT-4 reached a precision of 0.99, with follow-up evaluation showing GPT-4.1 achieving F1=0.94.
This paper addresses the challenge of Named Entity Recognition (NER) for person names within the specialized domain of Russian news texts concerning cultural events. The study utilizes the unique SPbLitGuide dataset, a collection of event announcements from Saint Petersburg spanning 1999 to 2019. A comparative evaluation of diverse NER models is presented, encompassing established transformer-based architectures such as DeepPavlov, RoBERTa, and SpaCy, alongside recent Large Language Models (LLMs) including GPT-3.5, GPT-4, and GPT-4o. Key findings highlight the superior performance of GPT-4o when provided with specific prompting for JSON output, achieving an F1 score of 0.93. Furthermore, GPT-4 demonstrated the highest precision at 0.99. The research contributes to a deeper understanding of current NER model capabilities and limitations when applied to morphologically rich languages like Russian within the cultural heritage domain, offering insights for researchers and practitioners. Follow-up evaluation with GPT-4.1 (April 2025) achieves F1=0.94 for both simple and structured prompts, demonstrating rapid progress across model families and simplified deployment requirements.