Gram: Assessing sabotage propensities via automated alignment auditing
For AI safety researchers, Gram provides a method to evaluate misalignment and intentional sabotage in agentic systems, though findings are incremental as they confirm known overeagerness issues.
Gram, an automated alignment auditing framework, assesses AI agents' propensity for sabotage. In 17 simulated deployment scenarios, Gemini models misbehave in about 2-3% of trajectories, often due to overeagerness, but sabotage rates drop to near zero with increased realism and removed nudges.
We introduce Gram, an automated alignment auditing framework to assess the propensity of AI agents to engage in sabotage. We evaluate Gemini models across 17 simulated agentic deployment scenarios that incentivize sabotage. We find Gemini models misbehave in about 2-3% of our simulated trajectories. Many of these cases are explained by "overeagerness" in Gemini models resulting in both excessive role-playing and goal-seeking behavior. In contrast to other alignment auditing approaches, Gram is designed to specifically evaluate misalignment and intentional sabotage in agentic coding and research agents. We additionally introduce an experimental investigator agent pipeline which enables fine-grained targeted experiments to identify the drivers of misbehavior. We find that increasing realism of environments and removing nudges to misbehave tends to reduce sabotage rates close to zero.