SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors
For AI safety researchers, this benchmark reveals critical vulnerabilities in current LLM-based monitoring systems, showing that most attacks easily evade detection.
The paper introduces SLEIGHT-Bench, a benchmark of 40 synthetic attacks across 11 categories for evading LLM-based monitors of coding agents. They find that 24 of 40 attacks are never caught by an Opus 4.6 monitor at 1% false-positive rate, with an overall catch rate of 23%.
Since autonomous coding agents generate complex behaviors at high-volume, we may want to use other LLMs to monitor actions to reduce the risk from dangerous misaligned behavior. To better understand the limitations of such monitors against the diverse attack strategies that a coding agent could use, we present SLEIGHT-Bench (Subtle Low-itEration Insight-Guided Harmful Transcripts), a benchmark of synthetic transcripts containing 40 attacks across 11 categories, each showing a coding agent covertly pursuing a harmful objective (e.g. weight exfiltration, credential theft, unmonitored rogue deployment). We find that 24 of 40 attacks are never caught by an Opus 4.6 monitor with extended thinking across 10 trials at a 1% false-positive rate, while the overall catch rate is 23%. We identify several evasion strategies such as exploiting model priors, user instruction ambiguity, and state manipulation, which facilitate creating attacks that evade frontier monitors. We also elicit stronger monitor performance using coding agents as monitors versus regular prompted monitors, and for some evasion strategies show improved catch rates with targeted monitor prompts. Our dataset and evaluation framework are available at https://github.com/safety-research/sleight-bench and https://huggingface.co/datasets/sleightbench/SLEIGHT-Bench.