PG-Agent: An Agent Powered by Page Graph
This work addresses a bottleneck in GUI automation for developers and users by enhancing agent generalization, though it is incremental as it builds on existing MLLM and RAG techniques.
The paper tackles the problem of GUI agents struggling to generalize to new scenarios due to insufficient modeling of page transitions, by proposing PG-Agent, which uses page graphs and retrieval-augmented generation to improve performance, achieving state-of-the-art results on benchmarks with up to 20% higher success rates.
Graphical User Interface (GUI) agents possess significant commercial and social value, and GUI agents powered by advanced multimodal large language models (MLLMs) have demonstrated remarkable potential. Currently, existing GUI agents usually utilize sequential episodes of multi-step operations across pages as the prior GUI knowledge, which fails to capture the complex transition relationship between pages, making it challenging for the agents to deeply perceive the GUI environment and generalize to new scenarios. Therefore, we design an automated pipeline to transform the sequential episodes into page graphs, which explicitly model the graph structure of the pages that are naturally connected by actions. To fully utilize the page graphs, we further introduce Retrieval-Augmented Generation (RAG) technology to effectively retrieve reliable perception guidelines of GUI from them, and a tailored multi-agent framework PG-Agent with task decomposition strategy is proposed to be injected with the guidelines so that it can generalize to unseen scenarios. Extensive experiments on various benchmarks demonstrate the effectiveness of PG-Agent, even with limited episodes for page graph construction.