See, Think, Act: Teaching Multimodal Agents to Effectively Interact with GUI by Identifying Toggles
This addresses a key bottleneck in GUI interaction for multimodal agents, though it appears incremental as it builds on existing agents.
The paper tackled the problem of multimodal agents failing to reliably execute toggle control instructions in GUI environments, and proposed State-aware Reasoning (StaR), which improved toggle instruction execution accuracy by over 30%.
The advent of multimodal agents facilitates effective interaction within graphical user interface (GUI), especially in ubiquitous GUI control. However, their inability to reliably execute toggle control instructions remains a key bottleneck. To investigate this, we construct a state control benchmark with binary toggle instructions from public datasets. Evaluations of existing agents demonstrate their unreliability, particularly when the current toggle state already matches the desired state. To address the challenge, we propose State-aware Reasoning (StaR), a training method that teaches agents to perceive the current toggle state, analyze the desired state from the instruction, and act accordingly. Experiments on three multimodal agents demonstrate that StaR can improve toggle instruction execution accuracy by over 30\%. Further evaluations on three public benchmarks show that StaR also enhances general task performance. Finally, evaluations on a dynamic environment highlight the potential of StaR for real-world applications. Code, benchmark, and StaR-enhanced agents are available at https://github.com/ZrW00/StaR.