More Vulnerable than You Think: On the Stability of Tool-Integrated LLM Agents
This addresses a critical gap in real-world applicability for LLM agents by highlighting stability issues, though it is incremental in focusing on evaluation rather than solving the problem.
The study investigated the stability of tool-integrated LLM agents, finding they are highly vulnerable to errors at all stages of tool invocation, with open-source models more susceptible than proprietary ones, and model size increases not improving reasoning and potentially increasing vulnerability.
Current evaluations of tool-integrated LLM agents typically focus on end-to-end tool-usage evaluation while neglecting their stability. This limits their real-world applicability, as various internal or external factors can cause agents to crash or behave abnormally. Our research addresses this by investigating whether agents are vulnerable to errors throughout the entire tool invocation process, including reading tool documentation, selecting tools and generating parameters, and processing the tool's response. Through extensive experiments, we observe that agents are highly susceptible to errors at each stage and agents based on open-source models are more vulnerable than those based on proprietary models. We also find that increasing the model size does not significantly improve tool invocation reasoning and may make agents more vulnerable to attacks resembling normal user instructions. This highlights the importance of evaluating agent stability and offers valuable insights for future LLM development and evaluation.