Watson & Holmes: A Naturalistic Benchmark for Comparing Human and LLM Reasoning
This provides a more realistic benchmark for comparing AI and human reasoning, though it is incremental as it builds on existing game-based evaluation methods.
The authors tackled the problem of evaluating AI reasoning in naturalistic contexts by adapting the Watson & Holmes detective game as a benchmark, showing that AI model performance improved from the lower quartile to the top 5% of human performance over nine months in 2025.
Existing benchmarks for AI reasoning provide limited insight into how closely these capabilities resemble human reasoning in naturalistic contexts. We present an adaptation of the Watson & Holmes detective tabletop game as a new benchmark designed to evaluate reasoning performance using incrementally presented narrative evidence, open-ended questions and unconstrained language responses. An automated grading system was developed and validated against human assessors to enable scalable and replicable performance evaluation. Results show a clear improvement in AI model performance over time. Over nine months of 2025, model performance rose from the lower quartile of the human comparison group to approximately the top 5%. Around half of this improvement reflects steady advancement across successive model releases, while the remainder corresponds to a marked step change associated with reasoning-oriented model architectures. Systematic differences in the performance of AI models compared to humans, dependent on features of the specific detection puzzle, were mostly absent with the exception of a fall in performance for models when solving longer cases (case lengths being in the range of 1900-4000 words), and an advantage at inductive reasoning for reasoning models at early stages of case solving when evidence was scant.