Adversarial Reinforcement Learning for Large Language Model Agent Safety
This addresses security vulnerabilities in LLM agents for applications involving tool usage, though it is incremental as it builds on existing adversarial RL methods.
The paper tackles the problem of indirect prompt injections in Large Language Model (LLM) agents using tools like Google Search, which pose security risks such as data leakage, and proposes ARLAS, an adversarial reinforcement learning framework that co-trains an attacker and a defender LLM, resulting in agents with a significantly lower attack success rate and improved task success rate on benchmarks like BrowserGym and AgentDojo.
Large Language Model (LLM) agents can leverage tools such as Google Search to complete complex tasks. However, this tool usage introduces the risk of indirect prompt injections, where malicious instructions hidden in tool outputs can manipulate the agent, posing security risks like data leakage. Current defense strategies typically rely on fine-tuning LLM agents on datasets of known attacks. However, the generation of these datasets relies on manually crafted attack patterns, which limits their diversity and leaves agents vulnerable to novel prompt injections. To address this limitation, we propose Adversarial Reinforcement Learning for Agent Safety (ARLAS), a novel framework that leverages adversarial reinforcement learning (RL) by formulating the problem as a two-player zero-sum game. ARLAS co-trains two LLMs: an attacker that learns to autonomously generate diverse prompt injections and an agent that learns to defend against them while completing its assigned tasks. To ensure robustness against a wide range of attacks and to prevent cyclic learning, we employ a population-based learning framework that trains the agent to defend against all previous attacker checkpoints. Evaluated on BrowserGym and AgentDojo, agents fine-tuned with ARLAS achieve a significantly lower attack success rate than the original model while also improving their task success rate. Our analysis further confirms that the adversarial process generates a diverse and challenging set of attacks, leading to a more robust agent compared to the base model.