What Do AI Agents Actually Change? An Empirical Taxonomy of Mutation Patterns in Performance-Improving Pull Requests
For researchers in search-based software engineering, this provides empirical evidence that AI agent code mutations differ substantially from human-written mutations, enabling more informed operator design.
The paper analyzes 1,254 performance-relevant code changes from AI agent pull requests, finding that name modification (37.0%), object creation (26.4%), and type change (22.7%) dominate, contrasting with prior genetic improvement corpora where no change accounted for 84%. Agent identity and target strategy are shown to be informative priors for search-based software engineering.
AI coding agents are black boxes: we cannot inspect how they generate code, but we can inspect what they change. This distinction matters for search-based software engineering (SBSE), where techniques such as genetic improvement (in the performance-optimisation application we study) depend on mutation operators that reflect how code is actually transformed. Fewer than 1% of the 33,596 agent PRs in AIDev-pop target performance, making each case a rare window into otherwise opaque agent behaviour. We classify 1,254 performance-relevant diff hunks from 216 of these PRs, spanning five agent systems, against the 18-category syntactic mutation taxonomy of Even-Mendoza et al. (2025) using a dual-LLM intersection pipeline. Three categories dominate: name modification (37.0%), object creation (26.4%), and type change (22.7%), a profile markedly different from prior GI corpora where no change accounted for 84%. Each agent's deployed system commits to a distinctive mutation vocabulary, and each performance strategy activates a largely disjoint category subset. Agent identity and target strategy are therefore informative priors that narrow the effective SBSE operator space. Replication package: https://github.com/5uper6rain/ssbse-challenge-2026