Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining
For researchers building computer-using agents, this paper provides a diagnostic study showing that while trajectory mining can produce readable skill structures, current approaches fail to translate into meaningful policy improvements.
The paper investigates whether skill libraries mined from GUI interaction trajectories can improve downstream policies for computer-using agents. The mined clusters achieve high purity (five of eight clusters ≥0.95 purity), but the resulting skill-aware policy yields only marginal gains (GRPO improves IW skill-step accuracy from 18.5% to 20.5%) and underperforms trivial baselines, indicating that current methods are insufficient for reliable cross-domain transfer.
Explicit skill libraries make computer-using agents easier to inspect, but it remains unclear whether such libraries can be mined from interaction data in a way that improves downstream policies. We study this question through a three-stage pipeline that segments GUI trajectories, clusters segments into candidate skills, and trains a skill-aware policy from the resulting annotations. The mined clusters are readable on the source benchmark: five of eight clusters have at least 0.95 purity against InteraSkill Workflows labels. However, readability does not imply transfer. GRPO improves IW skill-step accuracy only from 18.5\% to 20.5\%, leaves BrowseComp+ essentially unchanged, and underperforms trivial frequency priors on key source-domain metrics. We therefore present the method as a diagnostic study: trajectory mining can expose inspectable skill structure, but the current boundary detector, orderless segment representation, and offline reward model are insufficient for reliable cross-domain policy improvement.