MultiPragEval: Multilingual Pragmatic Evaluation of Large Language Models
This addresses the need for evaluating LLMs on higher-level language understanding, specifically pragmatic inference, for AI researchers and developers, though it is incremental as it extends existing evaluation frameworks to multilingual contexts.
The study introduced MultiPragEval, a multilingual pragmatic evaluation benchmark for LLMs across English, German, Korean, and Chinese, and found that Claude3-Opus significantly outperformed other models, establishing a state-of-the-art, with Solar-10.7B and Qwen1.5-14B as strong open-source competitors.
As the capabilities of Large Language Models (LLMs) expand, it becomes increasingly important to evaluate them beyond basic knowledge assessment, focusing on higher-level language understanding. This study introduces MultiPragEval, the first multilingual pragmatic evaluation of LLMs, designed for English, German, Korean, and Chinese. Comprising 1200 question units categorized according to Grice's Cooperative Principle and its four conversational maxims, MultiPragEval enables an in-depth assessment of LLMs' contextual awareness and their ability to infer implied meanings. Our findings demonstrate that Claude3-Opus significantly outperforms other models in all tested languages, establishing a state-of-the-art in the field. Among open-source models, Solar-10.7B and Qwen1.5-14B emerge as strong competitors. By analyzing pragmatic inference, we provide valuable insights into the capabilities essential for advanced language comprehension in AI systems.