Are Online Skill and Memory Modules Always Worth Their Tokens? A Budget-Constrained Study of Web Agents
For researchers and practitioners building online web agents, this work shows that the apparent gains from skill/memory modules often vanish when controlling for token budget, challenging the assumption that such modules are always beneficial.
Under a fixed total inference budget, a token-matched vanilla baseline (using the same budget for additional actor steps) matches or surpasses three augmentation methods (AWM, ASI, ReasoningBank) in aggregate success rate across three WebArena domains and three models, often using fewer total tokens. The effect extends to WorkArena-L1 with Qwen 3.6-27B.
Online web agents often augment a base actor with memory, workflow, or skill modules. These modules can improve performance, but they also consume test-time tokens, a cost rarely reported alongside the actor's inference cost. We study online augmentation, where this overhead is paid on every task, and re-evaluate its benefits under a fixed total inference budget. We compare AWM, ASI, and ReasoningBank with a token-matched vanilla baseline that uses the same budget for additional actor steps. Across three WebArena domains and three models, Gemini 3 Flash, GPT-5.4-mini, and Qwen 3.6-27B, the vanilla baseline matches or surpasses all three augmentation methods in aggregate success rate while often using fewer total tokens. We observe a similar trend on WorkArena-L1 with Qwen 3.6-27B, indicating that the effect extends to enterprise knowledge-work tasks. Our results suggest that skills and workflow memory can be useful in specific domains, but their apparent gains often vanish against a budget-matched actor. We further show that run-to-run variance materially affects outcomes and should be reported as a core evaluation criterion for online web agents.