Red-teaming Activation Probes using Prompted LLMs
This work addresses robustness issues for activation probes in high-stakes AI systems, offering a method to anticipate failures before deployment, though it is incremental as it builds on existing red-teaming and probing techniques.
The paper tackled the problem of unknown failure modes in activation probes under adversarial conditions by developing a lightweight black-box red-teaming procedure using prompted LLMs, which uncovered interpretable brittleness patterns like legalese-induced false positives and reduced but persistent vulnerabilities in a case study.
Activation probes are attractive monitors for AI systems due to low cost and latency, but their real-world robustness remains underexplored. We ask: What failure modes arise under realistic, black-box adversarial pressure, and how can we surface them with minimal effort? We present a lightweight black-box red-teaming procedure that wraps an off-the-shelf LLM with iterative feedback and in-context learning (ICL), and requires no fine-tuning, gradients, or architectural access. Running a case study with probes for high-stakes interactions, we show that our approach can help discover valuable insights about a SOTA probe. Our analysis uncovers interpretable brittleness patterns (e.g., legalese-induced FPs; bland procedural tone FNs) and reduced but persistent vulnerabilities under scenario-constraint attacks. These results suggest that simple prompted red-teaming scaffolding can anticipate failure patterns before deployment and might yield promising, actionable insights to harden future probes.