Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese
Provides a robust, bias-resistant evaluation method for LLMs in Brazilian Portuguese, addressing the need for reliable benchmarks in underrepresented languages.
Prosa introduces the first real user multi-turn Brazilian Portuguese chat benchmark with 1,000 conversations scored by three judges. Switching to binary rubric scoring with multi-judge filtering eliminates judge model bias, achieving full rank agreement across judges and increasing the average score gap between neighboring models by 47%.
Rankings produced by holistic LLM-as-a-judge scoring are sensitive to the bias of the chosen judge model. We show that switching to binary rubric scoring with multi-judge filtering removes this sensitivity: decomposing the judgement matters more than the judge model itself. To support this claim, we introduce Prosa, the first real user multi-turn Brazilian Portuguese chat benchmark: 1,000 WildChat conversations scored by three judges from three model families on 16 models. Under filtered rubric scoring the three judges agree on every one of the 16 ranks, whereas under holistic scoring they agree on only 7 of 16. Additionally, the rubric filtering pipeline increases the average score gap between neighbouring models by 47%, thereby improving Prosa's discriminative power. Evaluating a new model on Prosa costs approximately $2.1 when using Gemini 3 Flash as the judge. We release the benchmark and the filtering code to ensure that future models can be assessed under identical conditions. These artifacts also make our rubric-based scoring method reusable beyond Prosa, supporting other open-ended evaluation settings.