On the Robustness of GUI Grounding Models Against Image Attacks
This addresses the underexplored robustness challenges in GUI grounding models for intelligent agents, providing insights into vulnerabilities and establishing a benchmark for future research, though it is incremental as it focuses on evaluation rather than new solutions.
The study systematically evaluated the robustness of state-of-the-art GUI grounding models, such as UGround, against natural noise and adversarial attacks, finding they are highly sensitive to perturbations and low-resolution conditions across various GUI environments.
Graphical User Interface (GUI) grounding models are crucial for enabling intelligent agents to understand and interact with complex visual interfaces. However, these models face significant robustness challenges in real-world scenarios due to natural noise and adversarial perturbations, and their robustness remains underexplored. In this study, we systematically evaluate the robustness of state-of-the-art GUI grounding models, such as UGround, under three conditions: natural noise, untargeted adversarial attacks, and targeted adversarial attacks. Our experiments, which were conducted across a wide range of GUI environments, including mobile, desktop, and web interfaces, have clearly demonstrated that GUI grounding models exhibit a high degree of sensitivity to adversarial perturbations and low-resolution conditions. These findings provide valuable insights into the vulnerabilities of GUI grounding models and establish a strong benchmark for future research aimed at enhancing their robustness in practical applications. Our code is available at https://github.com/ZZZhr-1/Robust_GUI_Grounding.