GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks
This work addresses the need for collaborative GUI agents that assist users in complex software by understanding their intentions, rather than just automating actions, though it is incremental as it builds on prior automation-focused research.
The paper tackles the problem of GUI agents lacking understanding of user intent in open-ended tasks by introducing the GUIDE benchmark, which evaluates models on behavior state detection, intent prediction, and help prediction, finding that state-of-the-art models achieve only 44.6% and 55.0% accuracy on key tasks, but user context improves help prediction by up to 50.2 percentage points.
Graphical User Interface (GUI) agents have the potential to assist users in interacting with complex software (e.g., PowerPoint, Photoshop). While prior research has primarily focused on automating user actions through clicks and keystrokes, this paradigm overlooks human intention, where users value the ability to explore, iterate, and refine their ideas while maintaining agency. To move beyond automation and toward collaboration, GUI agents must understand what users are doing and why. We introduce GUIDE (GUI User Intent Detection Evaluation), a benchmark that evaluates AI models on their ability to perceive user behavior, infer intent, and provide assistance in open-ended GUI tasks. GUIDE consists of 67.5 hours of screen recordings from 120 novice user demonstrations with think-aloud narrations, across 10 software. GUIDE defines three tasks - (i) Behavior State Detection, (ii) Intent Prediction, and (iii) Help Prediction that test a model's ability to recognize behavior state, reason about goals, and decide when and how to help. Evaluations across eight state-of-the-art multimodal models reveal that all models struggled, achieving only 44.6% and 55.0% accuracy on behavior state and help prediction. However, providing user context significantly improved the performance, raising help prediction by up to 50.2pp, highlighting the critical role of structured user understanding in effective assistance. Our dataset is available at https://guide-bench.github.io.