Auto-scaling Continuous Memory for GUI Agent
This addresses the problem of scalable memory for GUI agents, enabling better generalization across interfaces and tasks, though it is incremental as it builds on existing VLM and agent frameworks.
The paper tackles the problem of GUI agents struggling with long-horizon tasks and unfamiliar interfaces by proposing a continuous memory system that encodes GUI trajectories into fixed-length embeddings, reducing context costs and preserving visual details. The result is improved success rates on real-world benchmarks, with Qwen-2.5-VL-7B + continuous memory achieving performance comparable to state-of-the-art closed-source models like GPT-4o and Claude-4.
We study how to endow GUI agents with scalable memory that help generalize across unfamiliar interfaces and long-horizon tasks. Prior GUI agents compress past trajectories into text tokens, which balloons context length and misses decisive visual cues (e.g., exact widget size and position). We propose a continuous memory that encodes each GUI trajectory into a fixed-length sequence of continuous embeddings using the VLM itself as an encoder; these embeddings are plugged directly into the backbone's input layer, sharply reducing context cost while preserving fine-grained visual information. As memory size and retrieval depth increase, performance improves monotonically, unlike text memories that degrade with long prompts. To grow memory at low cost, we introduce an auto-scaling data flywheel that (i) discovers new environments via search, (ii) synthesizes tasks with an open-source VLM, (iii) rolls out trajectories with the agent, and (iv) verifies success with the same VLM. Using this pipeline, we collect 100k+ trajectories for about \$4000 and fine-tune only the memory encoder (LoRA on a Q-Former, 1.2\% parameters) with 1,500 samples. On real-world GUI benchmarks, our memory-augmented agent consistently improves success rates under long horizons and distribution shifts. Notably, Qwen-2.5-VL-7B + continuous memory achieves performance comparable to state-of-the-art closed-source models (e.g., GPT-4o, Claude-4).