GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies
This addresses the problem of evaluating GUI agent robustness for researchers and developers, though it is incremental as it builds on existing dataset efforts.
The authors tackled the lack of realistic datasets for GUI agents by introducing GUI-Robust, a dataset incorporating seven common real-world anomalies, which revealed substantial performance degradation in state-of-the-art agents when tested on it.
The development of high-quality datasets is crucial for benchmarking and advancing research in Graphical User Interface (GUI) agents. Despite their importance, existing datasets are often constructed under idealized conditions, overlooking the diverse anomalies frequently encountered in real-world deployments. To address this limitation, we introduce GUI-Robust, a novel dataset designed for comprehensive GUI agent evaluation, explicitly incorporating seven common types of anomalies observed in everyday GUI interactions. Furthermore, we propose a semi-automated dataset construction paradigm that collects user action sequences from natural interactions via RPA tools and then generate corresponding step and task descriptions for these actions with the assistance of MLLMs. This paradigm significantly reduces annotation time cost by a factor of over 19 times. Finally, we assess state-of-the-art GUI agents using the GUI-Robust dataset, revealing their substantial performance degradation in abnormal scenarios. We anticipate that our work will highlight the importance of robustness in GUI agents and inspires more future research in this direction. The dataset and code are available at https://github.com/chessbean1/GUI-Robust..