AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI Agents
This addresses the problem of evaluating GUI agents for real-world mobile automation tasks, though it is incremental as it builds on existing benchmark limitations.
The paper tackles the lack of challenging evaluation benchmarks for mobile GUI agents by introducing AndroidLens, a framework with 571 long-latency tasks requiring over 26 steps on average, and finds that even the best models achieve only a 12.7% task success rate and 50.47% average task progress.
Graphical user interface (GUI) agents can substantially improve productivity by automating frequently executed long-latency tasks on mobile devices. However, existing evaluation benchmarks are still constrained to limited applications, simple tasks, and coarse-grained metrics. To address this, we introduce AndroidLens, a challenging evaluation framework for mobile GUI agents, comprising 571 long-latency tasks in both Chinese and English environments, each requiring an average of more than 26 steps to complete. The framework features: (1) tasks derived from real-world user scenarios across 38 domains, covering complex types such as multi-constraint, multi-goal, and domain-specific tasks; (2) static evaluation that preserves real-world anomalies and allows multiple valid paths to reduce bias; and (3) dynamic evaluation that employs a milestone-based scheme for fine-grained progress measurement via Average Task Progress (ATP). Our evaluation indicates that even the best models reach only a 12.7% task success rate and 50.47% ATP. We also underscore key challenges in real-world environments, including environmental anomalies, adaptive exploration, and long-term memory retention.