Hybrid Self-evolving Structured Memory for GUI Agents
This addresses the problem of inefficient memory retrieval for GUI agents, offering a novel solution with significant performance gains, though it is incremental in advancing existing memory approaches.
The paper tackles the challenge of enabling GUI agents to handle long-horizon computer-use tasks by proposing HyMEM, a graph-based memory system that improves performance, boosting Qwen2.5-VL-7B by +22.5% and outperforming models like Gemini2.5-Pro-Vision and GPT-4o.
The remarkable progress of vision-language models (VLMs) has enabled GUI agents to interact with computers in a human-like manner. Yet real-world computer-use tasks remain difficult due to long-horizon workflows, diverse interfaces, and frequent intermediate errors. Prior work equips agents with external memory built from large collections of trajectories, but relies on flat retrieval over discrete summaries or continuous embeddings, falling short of the structured organization and self-evolving characteristics of human memory. Inspired by the brain, we propose Hybrid Self-evolving Structured Memory (HyMEM), a graph-based memory that couples discrete high-level symbolic nodes with continuous trajectory embeddings. HyMEM maintains a graph structure to support multi-hop retrieval, self-evolution via node update operations, and on-the-fly working-memory refreshing during inference. Extensive experiments show that HyMEM consistently improves open-source GUI agents, enabling 7B/8B backbones to match or surpass strong closed-source models; notably, it boosts Qwen2.5-VL-7B by +22.5% and outperforms Gemini2.5-Pro-Vision and GPT-4o.