SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematics
For researchers and developers of retrieval systems for mathematical tasks, this benchmark addresses the lack of math-specific evaluation, showing that general-purpose benchmarks are insufficient.
SABER-Math is the first fully automated benchmark for evaluating mathematical information retrieval without expert annotation, built from 283K high-school math problems. It reveals that modern embedding models outperform classical and math-specific baselines but still struggle in symbol-heavy domains like Algebra and Calculus, and that general-purpose benchmarks like MTEB do not reliably predict mathematical retrieval performance.
As agentic AI systems tackle more complex mathematical tasks, they increasingly rely on information retrieval (IR) to search problem databases, theorem libraries, and educational resources. However, choosing the right retriever remains difficult, as it is infeasible to directly isolate its effect on downstream performance. On the other hand, existing retrieval-specific benchmarks often fail to capture fine-grained mathematical relevance, penalizing relevant documents. We address this gap by introducing SABER-Math, the first fully automated benchmark for evaluating mathematical IR without expert annotation. Starting from 283K high-school-level math problems with solutions, SABER-Math builds challenging reranking tasks in three steps: (i) first, LLMs extract concise solution summaries and mathematical topics for each problem; (ii) then, per-query relevant documents are discovered using ontology topic-based and lexical solutions-summary-based similarities, and (iii) finally, a Swiss-style LLM preference tournament produces fine-grained relevance ratings for the documents. We evaluate lexical retrievers, specialized mathematical retrieval systems, and recent embedding models. We find that while modern embedding models substantially outperform classical and math-specific baselines, even the strongest systems struggle in symbol-heavy domains like Algebra and Calculus. Importantly, we show that general-purpose IR benchmarks such as MTEB do not reliably predict mathematical performance, especially for recent embedding models, highlighting the need for math-specific retrieval benchmarks.