Modular and Multi-Path-Aware Offline Benchmarking for Mobile GUI Agents
This work addresses the need for scalable and reproducible evaluation methods for mobile GUI agents, which is crucial for developers and researchers in human-computer interaction, though it is incremental as it builds on existing benchmarking approaches.
The paper tackles the problem of evaluating mobile GUI agents by addressing limitations in current benchmarks, such as unfair penalization of alternative actions and lack of component-level analysis, and introduces MobiBench, a modular and multi-path-aware offline framework that achieves 94.72% agreement with human evaluators while maintaining scalability and reproducibility.
Mobile GUI Agents, AI agents capable of interacting with mobile applications on behalf of users, have the potential to transform human computer interaction. However, current evaluation practices for GUI agents face two fundamental limitations. First, they either rely on single path offline benchmarks or online live benchmarks. Offline benchmarks using static, single path annotated datasets unfairly penalize valid alternative actions, while online benchmarks suffer from poor scalability and reproducibility due to the dynamic and unpredictable nature of live evaluation. Second, existing benchmarks treat agents as monolithic black boxes, overlooking the contributions of individual components, which often leads to unfair comparisons or obscures key performance bottlenecks. To address these limitations, we present MobiBench, the first modular and multi path aware offline benchmarking framework for mobile GUI agents that enables high fidelity, scalable, and reproducible evaluation entirely in offline settings. Our experiments demonstrate that MobiBench achieves 94.72 percent agreement with human evaluators, on par with carefully engineered online benchmarks, while preserving the scalability and reproducibility of static offline benchmarks. Furthermore, our comprehensive module level analysis uncovers several key insights, including a systematic evaluation of diverse techniques used in mobile GUI agents, optimal module configurations across model scales, the inherent limitations of current LFMs, and actionable guidelines for designing more capable and cost efficient mobile agents.