CoAdapt-GUI: Joint Workflow Context and Policy Adaptation for Unseen GUI Applications
For developers of mobile GUI agents, this work addresses the challenge of generalizing to unseen apps with limited interaction budget, showing significant gains over a policy-only baseline.
CoAdapt-GUI introduces a test-time adaptation framework for mobile GUI agents that jointly adapts workflow context and policy to unseen applications without target demonstrations. It achieves 45.0% on AndroidWorld-Generalization (vs 37.5% for a policy-only baseline) and improves AndroidWorld Plus from 38.6% to 52.9%.
Mobile GUI agents remain brittle when deployed to applications absent from source training. We study novel-app generalization under a limited target interaction budget and without target demonstrations. We introduce CoAdapt-GUI, a test-time adaptation (TTA) framework that jointly adapts structured workflow context and policy from the agent's own target-app rollouts and rewards. The workflow context retains transferable procedures, failure modes, and verification rules while excluding app-bound source details. This separation allows reusable workflow knowledge to guide adaptation without transferring source-interface state. For policy adaptation, task-context-matched group-relative optimization updates a LoRA adapter on a frozen vision-language model. Across two unseen-app evaluations, CoAdapt-GUI reaches 45.0% on AndroidWorld-Generalization, compared with 37.5% for the reported Policy-Only TTA baseline, and raises AndroidWorld Plus performance from 38.6% to 52.9%. These results show that transfer-constrained workflow context provides substantial gains and that joint policy adaptation further improves held-out performance.