Automatic Red Teaming LLM-based Agents with Model Context Protocol Tools
This addresses security vulnerabilities in widely used LLM-based agents for developers and users, though it is incremental as it builds on prior proof-of-concept studies.
The paper tackles the risk of tool poisoning attacks on LLM-based agents using model context protocol (MCP) tools by proposing AutoMalTool, an automated red teaming framework that generates malicious MCP tools, which effectively manipulate mainstream agents and evade detection, revealing new security risks.
The remarkable capability of large language models (LLMs) has led to the wide application of LLM-based agents in various domains. To standardize interactions between LLM-based agents and their environments, model context protocol (MCP) tools have become the de facto standard and are now widely integrated into these agents. However, the incorporation of MCP tools introduces the risk of tool poisoning attacks, which can manipulate the behavior of LLM-based agents. Although previous studies have identified such vulnerabilities, their red teaming approaches have largely remained at the proof-of-concept stage, leaving the automatic and systematic red teaming of LLM-based agents under the MCP tool poisoning paradigm an open question. To bridge this gap, we propose AutoMalTool, an automated red teaming framework for LLM-based agents by generating malicious MCP tools. Our extensive evaluation shows that AutoMalTool effectively generates malicious MCP tools capable of manipulating the behavior of mainstream LLM-based agents while evading current detection mechanisms, thereby revealing new security risks in these agents.