scBench: Evaluating AI Agents on Single-Cell RNA-seq Analysis
This addresses the bottleneck of data analysis for research groups using single-cell RNA sequencing, though it is incremental as it builds on existing benchmarking efforts like SpatialBench.
The authors tackled the problem of evaluating AI agents' ability to analyze messy, real-world single-cell RNA sequencing datasets by introducing scBench, a benchmark of 394 verifiable problems, and found that accuracy across eight frontier models ranged from 29-53%, with platform choice affecting accuracy as much as model choice, causing drops of over 40 percentage points on less-documented technologies.
As single-cell RNA sequencing datasets grow in adoption, scale, and complexity, data analysis remains a bottleneck for many research groups. Although frontier AI agents have improved dramatically at software engineering and general data analysis, it remains unclear whether they can extract biological insight from messy, real-world single-cell datasets. We introduce scBench, a benchmark of 394 verifiable problems derived from practical scRNA-seq workflows spanning six sequencing platforms and seven task categories. Each problem provides a snapshot of experimental data immediately prior to an analysis step and a deterministic grader that evaluates recovery of a key biological result. Benchmark data on eight frontier models shows that accuracy ranges from 29-53%, with strong model-task and model-platform interactions. Platform choice affects accuracy as much as model choice, with 40+ percentage point drops on less-documented technologies. scBench complements SpatialBench to cover the two dominant single-cell modalities, serving both as a measurement tool and a diagnostic lens for developing agents that can analyze real scRNA-seq datasets faithfully and reproducibly.