PersLitEval: Fine-grained Benchmark and Evaluation of LLMs on Persian Literature Questions
This work provides a fine-grained evaluation of LLMs on Persian literature, revealing category-level performance disparities and failure modes that can guide targeted improvements for non-English literary understanding.
The authors introduce PersLitEval, a benchmark of 4,514 Persian literature multiple-choice questions, and evaluate six LLMs across ten prompting strategies, finding that models perform well on conceptual tasks but struggle with formal linguistic analysis, with spelling and word formation being the hardest categories.
Despite impressive multilingual capabilities, large language models (LLMs) remain poorly evaluated on literary knowledge in non-English languages. We introduce PersLitEval, a benchmark of 4,514 Persian literature multiple-choice questions across eight fine-grained categories spanning spelling, literary devices, grammar, vocabulary, word formation, and conceptual understanding, sourced from materials for the Konkur university entrance examination. We evaluate six LLMs across ten prompting strategies, revealing striking category-level disparities across three tiers of task difficulty: models reach higher accuracy on conceptual similarity tasks but struggle with formal linguistic analysis, with spelling and word formation proving the hardest across all models. Prompting strategy has a significant impact on performance, with explained few-shot examples yielding the best results, particularly on formal linguistic categories. An error analysis identifies three failure modes: semantic comprehension gaps, formal linguistic knowledge gaps, and counting/enumeration errors, suggesting that different categories require different improvement strategies.