GUI agent: Guided Exploration of User-Sensitive Screens
For developers of LLM-driven GUI agents, this work provides a method to detect user-sensitive states, improving safety and reliability in real-world deployment.
The paper addresses the problem of LLM agents encountering user-sensitive information in GUI environments. It develops an explorer agent that systematically identifies queries leading to user-sensitive states, enabling safe handover to users.
LLM agents are increasingly being used to automate tasks for users within an open GUI environment. They inevitably encounter screens containing user-sensitive information, for which takeover of task execution by the user is highly desirable or even necessary. State-of-the-art LLM-driven agents are usually fine-tuned to complete tasks regardless of the safety implications of their actions. This makes their real-world deployment difficult and adversely affects the reliability. Therefore, it is crucial to identify and categorize user-sensitive states and define user-sensitive queries. This dataset would be to engineers to recognize and request handover to the user in critical scenarios. This short paper develops an explorer agent that systematically explores the query space starting from one demonstrated task to identify queries that, if executed, would lead to user-sensitive states in a GUI environment.