APEX-SWE
This provides a benchmark for evaluating AI models in real-world software engineering tasks, addressing a gap in existing narrow evaluations, though it is incremental as it builds on prior benchmarking efforts.
The authors tackled the problem of assessing whether frontier AI models can perform economically valuable software engineering work by introducing the APEX-SWE benchmark, which includes integration and observability tasks, and found that Gemini 3 Pro achieved a Pass@1 score of 25%.
We introduce the AI Productivity Index for Software Engineering (APEX-SWE), a benchmark for assessing whether frontier AI models can execute economically valuable software engineering work. Unlike existing evaluations that focus on narrow, well-defined tasks, APEX-SWE assesses two novel task types that reflect real-world software engineering work: (1) Integration tasks (n=100), which require constructing end-to-end systems across heterogeneous cloud primitives, business applications, and infrastructure-as-code services, and (2) Observability tasks (n=100), which require debugging production failures using telemetry signals such as logs and dashboards, as well as unstructured context. We evaluated eight frontier models on APEX-SWE. Gemini 3 Pro (Thinking = High) performs best, with a Pass@1 score of 25\%. Our analysis shows that strong performance is primarily driven by epistemic reasoning, defined as the ability to distinguish between assumptions and verified facts, combined with agency to resolve uncertainty prior to acting. We open-source the APEX-SWE evaluation harness and a dev set (n=50).