An Agentic Evaluation Framework for AI-Generated Scientific Code in PETSc
This addresses the problem of insufficient evaluation for AI-generated code in scientific computing, particularly for HPC libraries like PETSc, though it is incremental as it builds on existing agent paradigms.
The authors tackled the challenge of evaluating AI-generated scientific code in HPC by introducing petscagent-bench, an agentic framework that assesses code across five categories, revealing that current models often fail on library-specific conventions despite generating readable code.
While large language models have significantly accelerated scientific code generation, comprehensively evaluating the generated code remains a major challenge. Traditional benchmarks reduce evaluation to test-case matching, an approach insufficient for library code in HPC where solver selection, API conventions, memory management, and performance are just as critical as functional correctness. To address this gap, we introduce petscagent-bench, an agentic framework built on an agents-evaluating-agents paradigm. Instead of relying on static scripts, petscagent-bench deploys a tool-augmented evaluator agent that compiles, executes, and measures code produced by a separate model-under-test agent, orchestrating a 14-evaluator pipeline across five scoring categories: correctness, performance, code quality, algorithmic appropriateness, and library-specific conventions. Because the agents communicate through standardized protocols (A2A and MCP), the framework enables black-box evaluation of any coding agent without requiring access to its source code. We demonstrate the framework on a benchmark suite of realistic problems using the PETSc library for HPC. Our empirical analysis of frontier models reveals that while current models generate readable, well-structured code, they consistently struggle with library-specific conventions that traditional pass/fail metrics completely miss.