ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agents
This addresses the problem of building efficient and generalizable GUI agents for automation tasks, representing a novel paradigm rather than an incremental improvement.
The paper tackles the problem of GUI agents struggling with generalization to varying screen resolutions and data scarcity by proposing ToolTok, a multi-step pathfinding approach that models operations as sequences of tool usage with learnable token embeddings. The method achieves superior performance among 4B-scale models and remains competitive with a 235B model while using less than 1% of the training data required by other approaches.
Existing GUI agent models relying on coordinate-based one-step visual grounding struggle with generalizing to varying input resolutions and aspect ratios. Alternatives introduce coordinate-free strategies yet suffer from learning under severe data scarcity. To address the limitations, we propose ToolTok, a novel paradigm of multi-step pathfinding for GUI agents, where operations are modeled as a sequence of progressive tool usage. Specifically, we devise tools aligned with human interaction habits and represent each tool using learnable token embeddings. To enable efficient embedding learning under limited supervision, ToolTok introduces a semantic anchoring mechanism that grounds each tool with semantically related concepts as natural inductive bias. To further enable a pre-trained large language model to progressively acquire tool semantics, we construct an easy-to-hard curriculum consisting of three tasks: token definition question-answering, pure text-guided tool selection, and simplified visual pathfinding. Extensive experiments on multiple benchmarks show that ToolTok achieves superior performance among models of comparable scale (4B) and remains competitive with a substantially larger model (235B). Notably, these results are obtained using less than 1% of the training data required by other post-training approaches. In addition, ToolTok demonstrates strong generalization across unseen scenarios. Our training & inference code is open-source at https://github.com/ZephinueCode/ToolTok.