Do Agents Know What They Can't Do? Evaluating Feasibility Awareness in Tool-Using Agents
For developers of tool-using agents, this work highlights a critical bottleneck in feasibility awareness and provides a benchmark to evaluate and improve early stopping in infeasible tasks.
The paper proposes FeasiGen, an automatic pipeline to create infeasible agent tasks by masking critical tools, and evaluates agents' ability to detect infeasibility. Results show agents have weak detection, with false continue rates up to 73.9%, but multi-agent architectures reduce errors.
Tool-using agents often incur substantial computational cost due to long reasoning chains and iterative tool usage. In practical scenarios, many tasks become infeasible under constrained tool environments, where the capabilities required for successful task completion are unavailable. Detecting infeasible tasks and stopping execution early can significantly reduce unnecessary execution cost. In this work, we propose FeasiGen, an automatic pipeline for constructing infeasible agent tasks by identifying the critical tools required for successful task completion. Our approach extracts tool-calling traces from successful executions across multiple agent systems, identifies critical tools consistently shared across diverse execution strategies, and masks these tools to automatically transform solvable tasks into infeasible ones. Human verification confirms that the infeasibility annotations for our constructed tasks achieve over 94% accuracy. We further introduce feasibility-aware evaluation metrics for measuring whether agents can recognize infeasible tasks and stop execution appropriately. Extensive evaluations across nine models reveal substantially weak infeasibility detection ability, with false continue rate reaching up to 73.9%. We further observe that multi-agent architectures significantly reduce erroneous execution under infeasible conditions.