Office Comprehension Benchmark
This benchmark addresses the need for standardized evaluation of LLMs on real-world office document comprehension, revealing significant performance gaps even in top models.
The paper introduces Office Comprehension Bench (OCB), the first public benchmark for evaluating LLMs on Word, Excel, and PowerPoint comprehension across file fidelity and domain-specific reasoning. The best frontier system achieves only 59.3% on Domain Q&A, with limited gains from deeper reasoning or higher-tier models.
We introduce Office Comprehension Bench (OCB), the first public benchmark to jointly evaluate LLM systems on Word, Excel, and PowerPoint comprehension over native file formats (.docx, .xlsx, .pptx) and their variants. OCB consists of two tracks. File Fidelity Q&A tests structural and visual perception of office artifacts - tables, charts, embedded images, formulas, and app-specific elements such as headers, speaker notes, and named ranges. Domain Q&A tests expert-level reasoning grounded in real-world industry documents across 12 professional domains, with queries requiring multi-step analysis and synthesis across documents. Each reference answer is decomposed into atomic, binary-gradable claims, and an ensemble of LLM judges scores responses against each claim independently. Even the strongest frontier system in its default reasoning mode reaches only about 59.3% on Domain Q&A; increasing thinking depth within a tier does not move performance materially, while moving to a higher product tier yields modest gains. We release the dataset, evaluation tooling, judge prompt, and a public leaderboard.