Zoom to Essence: Trainless GUI Grounding by Inferring upon Interface Elements
This addresses the challenge of efficient GUI interaction for users by reducing reliance on costly training data, though it is incremental as it builds on existing MLLM capabilities.
The paper tackles the problem of high data annotation costs and performance dependency in GUI agents by proposing ZoomUI, a trainless method that uses inference scaling to guide MLLMs in grounding natural language instructions to UI elements, achieving results that reach or surpass state-of-the-art baselines on benchmarks.
Multimodal Large Language Model (MLLM)-based Graphical User Interface (GUI) agents develop rapidly, with visual grounding that maps natural language instructions to target UI elements serving as the core capability. Existing GUI agents typically fine-tune MLLM on massive datasets to handle challenges in understanding instructions and UI interfaces, which not only incurs high data annotation costs but also makes performance dependent on data quality and distribution. To avoid such cumbersome yet ineffective training, we notice that complex UI interfaces can be decomposed into basic visual elements directly understandable by common MLLMs. Consequently, we propose ZoomUI that leverages inference scaling to guide common MLLMs in progressively anchor instruction elements to increasingly detailed interface elements. Specifically, ZoomUI first optimizes the latent thinking to transform original instruction into element visual features description, and subsequently leverages internal attention to iteratively zoom in target element interface region. Evaluations on extensive benchmarks demonstrate that ZoomUI reaches or even surpasses SOTA baselines.