FIBER: A Multilingual Evaluation Resource for Factual Inference Bias
This addresses concerns about factual inference bias in LLMs for multilingual applications, though it is incremental as it extends existing benchmarks to multilingual and multi-entity contexts.
The paper tackles the problem of evaluating factual reliability and biases in large language models by introducing FIBER, a multilingual benchmark for factual knowledge in single- and multi-entity settings, finding that prompt language influences entity selection bias (e.g., 31% of topics show bias >0.5) and models struggle more with multi-entity questions, with larger models performing better.
Large language models are widely used across domains, yet there are concerns about their factual reliability and biases. Factual knowledge probing offers a systematic means to evaluate these aspects. Most existing benchmarks focus on single-entity facts and monolingual data. We therefore present FIBER, a multilingual benchmark for evaluating factual knowledge in single- and multi-entity settings. The dataset includes sentence completion, question-answering, and object-count prediction tasks in English, Italian, and Turkish. Using FIBER, we examine whether the prompt language induces inference bias in entity selection and how large language models perform on multi-entity versus single-entity questions. The results indicate that the language of the prompt can influence the model's generated output, particularly for entities associated with the country corresponding to that language. However, this effect varies across different topics such that 31% of the topics exhibit factual inference bias score greater than 0.5. Moreover, the level of bias differs across languages such that Turkish prompts show higher bias compared to Italian in 83% of the topics, suggesting a language-dependent pattern. Our findings also show that models face greater difficulty when handling multi-entity questions than the single-entity questions. Model performance differs across both languages and model sizes. The highest mean average precision is achieved in English, while Turkish and Italian lead to noticeably lower scores. Larger models, including Llama-3.1-8B and Qwen-2.5-7B, show consistently better performance than smaller 3B-4B models.