OEP: Poisoning Self-Evolving LLM Agents via Locally Correct but Non-Transferable Experiences
This work exposes a new security vulnerability in memory-augmented LLM agents for developers and deployers, though the attack is domain-specific and requires crafting clean edge-cases.
OEP is a low-privilege black-box attack that poisons self-evolving LLM agents by injecting locally correct but non-transferable experiences, achieving attack success rates above 50% on GPT-4o agents across three domains and outperforming existing attacks under LLM auditing defense.
Memory-augmented large language model (LLM) agents use iterative reflection and self-evolution to solve complex tasks, but these mechanisms introduce security risks. Existing agentic memory attacks require privileged access or explicit malicious content, making them detectable by advanced safety filters. This leaves a subtler attack surface underexplored: whether adversaries can induce agent to generate experiences that appear locally correct and semantically plausible yet induce harmful generalization during reflection. We find that reflective agents are vulnerable to such clean experiences, especially when paired with severe but plausible hypothetical consequences. Based on this observation, we introduce Obsessive Experience Poisoning (OEP), a low-privilege black-box attack requiring no direct control over the system prompt or memory database. OEP constructs adversarial clean edge-cases that combine locally correct solutions, non-transferable methods, and severe consequences, biasing reflection toward risk-averse rule formation. During memory consolidation, agents may over-trust self-generated reflections and distill localized experiences into high-priority but over-generalized rules, causing downstream failures. Evaluations across three domains show that OEP achieves ASR above 50\% with GPT-4o agents, and outperforms existing attacks under LLM auditing defense.