Instruction Agent: Enhancing Agent with Expert Demonstration
This work addresses the problem of unreliable GUI task automation for users, representing an incremental improvement by integrating expert demonstrations into existing agent frameworks.
The paper tackles the challenge of GUI agents struggling with complex tasks involving novel UI elements and long-horizon actions by introducing Instruction Agent, which leverages expert demonstrations to achieve a 60% success rate on tasks where top-ranked agents previously failed.
Graphical user interface (GUI) agents have advanced rapidly but still struggle with complex tasks involving novel UI elements, long-horizon actions, and personalized trajectories. In this work, we introduce Instruction Agent, a GUI agent that leverages expert demonstrations to solve such tasks, enabling completion of otherwise difficult workflows. Given a single demonstration, the agent extracts step-by-step instructions and executes them by strictly following the trajectory intended by the user, which avoids making mistakes during execution. The agent leverages the verifier and backtracker modules further to improve robustness. Both modules are critical to understand the current outcome from each action and handle unexpected interruptions(such as pop-up windows) during execution. Our experiments show that Instruction Agent achieves a 60% success rate on a set of tasks in OSWorld that all top-ranked agents failed to complete. The Instruction Agent offers a practical and extensible framework, bridging the gap between current GUI agents and reliable real-world GUI task automation.