XTRUST: On the Multilingual Trustworthiness of Large Language Models
This addresses the need for assessing LLM trustworthiness in global deployments, particularly for sensitive applications, though it is incremental as it extends existing single-language benchmarks to a multilingual context.
The authors tackled the problem of evaluating the trustworthiness of large language models (LLMs) across multiple languages by introducing XTRUST, the first comprehensive multilingual benchmark covering 10 languages and diverse topics like toxicity and fairness, and found that many LLMs struggle with low-resource languages such as Arabic and Russian.
Large language models (LLMs) have demonstrated remarkable capabilities across a range of natural language processing (NLP) tasks, capturing the attention of both practitioners and the broader public. A key question that now preoccupies the AI community concerns the capabilities and limitations of these models, with trustworthiness emerging as a central issue, particularly as LLMs are increasingly applied in sensitive fields like healthcare and finance, where errors can have serious consequences. However, most previous studies on the trustworthiness of LLMs have been limited to a single language, typically the predominant one in the dataset, such as English. In response to the growing global deployment of LLMs, we introduce XTRUST, the first comprehensive multilingual trustworthiness benchmark. XTRUST encompasses a diverse range of topics, including illegal activities, hallucination, out-of-distribution (OOD) robustness, physical and mental health, toxicity, fairness, misinformation, privacy, and machine ethics, across 10 different languages. Using XTRUST, we conduct an empirical evaluation of the multilingual trustworthiness of five widely used LLMs, offering an in-depth analysis of their performance across languages and tasks. Our results indicate that many LLMs struggle with certain low-resource languages, such as Arabic and Russian, highlighting the considerable room for improvement in the multilingual trustworthiness of current language models. The code is available at https://github.com/LluckyYH/XTRUST.