Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents
This reveals critical vulnerabilities in current defenses for LLM agents, highlighting the need for adaptive attack evaluation to ensure robustness, though it is incremental as it builds on existing defense testing.
The paper tackled the problem of insufficient robustness in defenses against indirect prompt injection attacks on LLM agents by evaluating eight defenses and bypassing all of them with adaptive attacks, achieving over 50% attack success rate.
Large Language Model (LLM) agents exhibit remarkable performance across diverse applications by using external tools to interact with environments. However, integrating external tools introduces security risks, such as indirect prompt injection (IPI) attacks. Despite defenses designed for IPI attacks, their robustness remains questionable due to insufficient testing against adaptive attacks. In this paper, we evaluate eight different defenses and bypass all of them using adaptive attacks, consistently achieving an attack success rate of over 50%. This reveals critical vulnerabilities in current defenses. Our research underscores the need for adaptive attack evaluation when designing defenses to ensure robustness and reliability. The code is available at https://github.com/uiuc-kang-lab/AdaptiveAttackAgent.