An Analysis of Multilingual FActScore
This work addresses the problem of unreliable factuality assessment for multilingual text generation, which is incremental as it extends an existing metric to new languages.
This paper analyzed the limitations of the FActScore metric for evaluating factuality in multilingual long-form texts generated by LLMs, finding that no LLM produced consistent scores across languages and that using Wikipedia as a knowledge source hindered accuracy, especially in low-resource languages, with proposed mitigations improving estimation.
FActScore has gained popularity as a metric to estimate the factuality of long-form texts generated by Large Language Models (LLMs) in English. However, there has not been any work in studying the behavior of FActScore in other languages. This paper studies the limitations of each component in the four-component pipeline of FActScore in the multilingual setting. We introduce a new dataset for FActScore on texts generated by strong multilingual LLMs. Our evaluation shows that LLMs exhibit distinct behaviors in both fact extraction and fact scoring tasks. No LLM produces consistent and reliable FActScore across languages with varying levels of resources. We also find that the knowledge source plays an important role in the quality of the estimated FActScore. Using Wikipedia as the knowledge source may hinder the true FActScore of long-form text due to its limited coverage in medium- and low-resource languages. We also incorporate three mitigations to our knowledge source that ultimately improve FActScore estimation across all languages.