TARAZ: Persian Short-Answer Question Benchmark for Cultural Evaluation of Language Models
This work provides the first standardized benchmark for measuring cultural understanding in Persian, addressing the limitations of existing English-centric and multiple-choice benchmarks for researchers and developers working with Persian LLMs.
This paper introduces TARAZ, a new benchmark for evaluating the cultural competence of large language models in Persian using a short-answer format. Their hybrid evaluation method, combining rule-based morphological normalization with syntactic and semantic similarity, improves scoring consistency by 10% compared to exact-match baselines.
This paper presents a comprehensive evaluation framework for assessing the cultural competence of large language models (LLMs) in Persian. Existing Persian cultural benchmarks rely predominantly on multiple-choice formats and English-centric metrics that fail to capture Persian's morphological complexity and semantic nuance. Our framework introduces a Persian-specific short-answer evaluation that combines rule-based morphological normalization with a hybrid syntactic and semantic similarity module, enabling robust soft-match scoring beyond exact string overlap. Through systematic evaluation of 15 state-of-the-art open- and closed-source models, we demonstrate that our hybrid evaluation improves scoring consistency by +10% compared to exact-match baselines by capturing meaning that surface-level methods cannot detect. We publicly release our evaluation framework, providing the first standardized benchmark for measuring cultural understanding in Persian and establishing a reproducible foundation for cross-cultural LLM evaluation research.