AIAug 6, 2025

GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning

arXiv:2508.04389v16 citationsh-index: 17
Originality Incremental advance
AI Analysis

This addresses the data and cost inefficiency of supervised fine-tuning for GUI agents, offering a more efficient alternative for developers and researchers in human-computer interaction.

The paper tackles the problem of GUI visual grounding by proposing GuirlVG, a reinforcement learning-based method that uses only 5.2K training samples to outperform supervised fine-tuning methods trained on over 10M samples, achieving improvements of 7.7% on ScreenSpot, 17.2% on ScreenSpotPro, and 91.9% accuracy on ScreenSpotV2.

Graphical user interface visual grounding (GUI-VG), a core capability for GUI agents, has primarily relied on supervised fine-tuning (SFT) of multimodal large language models (MLLMs), which demands extensive data curation and significant training costs. However, as MLLMs continue to advance and even cover GUI domains during pretraining, the necessity of exhaustive SFT post-training becomes increasingly questionable. Meanwhile, recent successes of rule-based reinforcement fine-tuning (RFT) suggest a more efficient alternative. Despite this promise, the optimal manner of applying RFT for GUI-VG remains unexplored. To bridge this gap, we introduce GuirlVG, a reinforcement learning-based GUI-VG method built on a systematic empirical study and a novel stabilization technique. We find that naive application of RFT underperforms the SFT baseline, motivating a deeper exploration. First, we decompose RFT into its core components and analyze the optimal formulation of each. Second, we propose a novel Adversarial KL Factor that dynamically stabilizes training to mitigate reward over-optimization. Third, we further explore the training configurations of RFT to enhance effectiveness. Extensive experiments show that GuirlVG, with only 5.2K training samples, outperforms SFT methods trained on over 10M samples, achieving a 7.7% improvement on ScreenSpot, a 17.2% improvement on ScreenSpotPro, and 91.9% accuracy on ScreenSpotV2.

Foundations

The foundational work for this paper's niche, ranked by how specifically the neighbourhood builds on it — not by global fame.

Your Notes