PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguese
This work addresses the need for systematic evaluation of LLMs in Portuguese, providing a foundational benchmark for future research in this domain-specific context.
The authors tackled the problem of evaluating large language models (LLMs) in Portuguese by introducing PoETa v2, a benchmark with over 40 tasks, and assessed more than 20 models, revealing how computational investment and language-specific adaptation affect performance compared to English.
Large Language Models (LLMs) exhibit significant variations in performance across linguistic and cultural contexts, underscoring the need for systematic evaluation in diverse languages. In this work, we present the most extensive evaluation of LLMs for the Portuguese language to date. Leveraging our newly introduced PoETa v2 benchmark -- a comprehensive suite of over 40 tasks in Portuguese -- we assess more than 20 models covering a broad spectrum of training scales and computational resources. Our study reveals how computational investment and language-specific adaptation impact performance in Portuguese, while also analyzing performance gaps in comparison to equivalent tasks in English. Through this benchmark and analysis, PoETa v2 lays the groundwork for future research on Portuguese language modeling and evaluation. The benchmark is available at https://github.com/PoETaV2/PoETaV2.