MARCA: A Checklist-Based Benchmark for Multilingual Web Search
For researchers evaluating LLMs in multilingual web search, this benchmark fills a gap for Portuguese, but the contribution is incremental as it extends existing evaluation paradigms.
MARCA is a bilingual (English and Portuguese) benchmark with 52 multi-entity questions and checklist rubrics to evaluate LLMs on web-based information seeking. Orchestration improved coverage, but large performance differences and variability in English-to-Portuguese transfer were observed.
Large language models (LLMs) are increasingly used as sources of information, yet their reliability depends on the ability to search the web, select relevant evidence, and synthesize complete answers. While recent benchmarks evaluate web-browsing and agentic tool use, multilingual settings, and Portuguese in particular, remain underexplored. We present \textsc{MARCA}, a bilingual (English and Portuguese) benchmark for evaluating LLMs on web-based information seeking. \textsc{MARCA} consists of 52 manually authored multi-entity questions, paired with manually validated checklist-style rubrics that explicitly measure answer completeness and correctness. We evaluate 14 models under two interaction settings: a Basic framework with direct web search and scraping, and an Orchestrator framework that enables task decomposition via delegated subagents. To capture stochasticity, each question is executed multiple times and performance is reported with run-level uncertainty. Across models, we observe large performance differences, find that orchestration often improves coverage, and identify substantial variability in how models transfer from English to Portuguese. The benchmark is available at https://github.com/maritaca-ai/MARCA