ProfBench: Multi-Domain Rubrics requiring Professional Knowledge to Answer and Judge
This addresses the problem of limited evaluation methods for LLMs in real-world professional applications, making it accessible to the broader community, though it is incremental as it builds on existing benchmarking approaches.
The paper tackles the challenge of evaluating large language models (LLMs) in professional domains by introducing ProfBench, a dataset of over 7000 response-criterion pairs evaluated by human experts across fields like Physics, Chemistry, Finance, and Consulting, which reveals that top models like GPT-5-high achieve only 65.9% overall performance.
Evaluating progress in large language models (LLMs) is often constrained by the challenge of verifying responses, limiting assessments to tasks like mathematics, programming, and short-form question-answering. However, many real-world applications require evaluating LLMs in processing professional documents, synthesizing information, and generating comprehensive reports in response to user queries. We introduce ProfBench: a set of over 7000 response-criterion pairs as evaluated by human-experts with professional knowledge across Physics PhD, Chemistry PhD, Finance MBA and Consulting MBA. We build robust and affordable LLM-Judges to evaluate ProfBench rubrics, by mitigating self-enhancement bias and reducing the cost of evaluation by 2-3 orders of magnitude, to make it fair and accessible to the broader community. Our findings reveal that ProfBench poses significant challenges even for state-of-the-art LLMs, with top-performing models like GPT-5-high achieving only 65.9\% overall performance. Furthermore, we identify notable performance disparities between proprietary and open-weight models and provide insights into the role that extended thinking plays in addressing complex, professional-domain tasks. Data: https://huggingface.co/datasets/nvidia/ProfBench and Code: https://github.com/NVlabs/ProfBench