GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning
This work addresses the labor-intensive and error-prone data collection for GUI agents, offering a scalable alternative for researchers and practitioners in human-computer interaction and automation.
The paper tackles the problem of training Vision-Language Models (VLMs) for GUI agents without relying on large annotated datasets by introducing a self-supervised inverse dynamics task and a reinforcement learning framework, resulting in up to an 11.2% increase in GUI automation accuracy across benchmarks.
Training effective Vision-Language Models (VLMs) for GUI agents typically depends on large-scale annotated datasets, whose collection is both labor-intensive and error-prone. We introduce K-step GUI Transition, a self-supervised inverse dynamics task in which VLMs learn GUI dynamics by predicting the initial action that causes a transition between two GUI states. This approach eliminates the need for natural language instructions and enables scalable dataset construction from existing GUI trajectories or automated exploration. Building on this task, we propose GUI-Shift, a reinforcement learning (RL) framework that combines rule-based optimization with data filtering to improve VLM performance. We conduct extensive experiments using multiple VLM backbones across four benchmarks, spanning GUI task automation (AndroidControl, GUI Odyssey) and GUI grounding (ScreenSpot-v2, ScreenSpot-Pro). Our results show that training on GUI-Shift generalizes well to both GUI automation and grounding tasks, yielding up to an 11.2% increase in GUI automation accuracy. This study underscores the potential of self-supervised RL to leverage unlabeled GUI trajectories and offers a scalable alternative to training with annotated samples.