Do LLMs Know Tool Irrelevance? Demystifying Structural Alignment Bias in Tool Invocations
Identifies and addresses a previously overlooked failure mode in LLM tool invocation, improving reliability for developers deploying LLMs with external tools.
LLMs exhibit a structural alignment bias, where they invoke tools based on parameter matching even when the tool is semantically irrelevant to the query. The authors introduce SABEval to measure this bias and propose a rebalancing strategy that reduces invocation errors without harming general tool-use performance.
Large language models (LLMs) have demonstrated impressive capabilities in utilizing external tools. In practice, however, LLMs are often exposed to tools that are irrelevant to the user's query, in which case the desired behavior is to refrain from invocations. In this work, we identify a widespread yet overlooked mechanistic flaw in tool refusal, which we term structural alignment bias: Even when a tool fails to serve the user's goal, LLMs still tend to invoke it whenever query attributes can be validly assigned to tool parameters. To systematically study this bias, we introduce SABEval, a new dataset that decouples structural alignment from semantic relevance. Our analysis shows that structural alignment bias induces severe tool-invocation errors in LLMs, yet remains largely unaccounted for in existing evaluations. To investigate the internal mechanisms underlying this bias, we propose Contrastive Attention Attribution, which reveals two competing pathways for semantic checking and structural matching. The relative strength of these pathways drives LLMs' tool invocation decisions. Based on these findings, we further introduce a rebalancing strategy that effectively mitigates structural alignment bias, as demonstrated by extensive experiments, without degrading general tool-use capabilities.