MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Users
This addresses the challenge of high-cost data collection for mobile automation, offering a scalable solution for real-world applications, though it appears incremental by combining existing techniques like auto-annotation and federated learning.
The paper tackles the problem of training mobile GUI agents by proposing MobileA3gent, a framework that uses decentralized self-sourced data from users to automatically collect datasets and enhance federated training, achieving superior performance at only 1% of the cost compared to traditional approaches.
The advancement of mobile GUI agents has opened new opportunities for automating tasks on mobile devices. Training these agents requires large-scale high-quality data, which is prohibitively expensive when relying on human labor. Given the vast population of global mobile phone users, if automated data collection from them becomes feasible, the resulting data volume and the subsequently trained mobile agents could reach unprecedented levels. Nevertheless, two major challenges arise: (1) extracting user instructions without human intervention and (2) utilizing distributed user data while preserving privacy. To tackle these challenges, we propose MobileA3gent, a collaborative framework that trains mobile GUI Agents using decentralized self-sourced data from diverse users. The framework comprises two components, each targeting a specific challenge: (1) Auto-Annotation, which enables the automatic collection of high-quality datasets during users' routine phone usage with minimal cost. (2) FedVLM-A, which enhances federated VLM training under non-IID distributions by incorporating adapted global aggregation based on both episode-level and step-level variability. Extensive experiments prove that MobileA3gent achieves superior performance over traditional approaches at only 1% of the cost, highlighting its potential for real-world applications