SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks
This addresses the problem of automatically learning skills for LLM agents in real-world tasks, though it is incremental as it benchmarks existing methods rather than proposing new ones.
The authors introduced SkillLearnBench, the first benchmark for evaluating continual learning methods for agent skill generation, comprising 20 real-world tasks across 15 sub-domains, and found that while all continual learning methods improved over baselines, no method consistently outperformed others across tasks or LLMs, with scaling to stronger LLMs not reliably helping.
Skills have become the de facto way to enable LLM agents to perform complex real-world tasks with customized instructions, workflows, and tools, but how to learn them automatically and effectively remains unclear. We introduce SkillLearnBench, the first benchmark for evaluating continual skill learning methods, comprising 20 verified, skill-dependent tasks across 15 sub-domains derived from a real-world skill taxonomy , evaluated at three levels: skill quality, execution trajectory, and task outcome. Using this benchmark, we evaluate recent continual learning techniques, those leveraging one-shot, self/teacher feedback, and skill creator to generate skills from agent experiences. We find that all continual learning methods improve over the no-skill baseline, yet consistent gains remain elusive: no method leads across all tasks and LLMs, and scaling to stronger LLMs does not reliably help. Continual learning improves tasks with clear, reusable workflows but struggles on open-ended tasks, and using stronger LLM backbones does not consistently produce better skills. Our analysis also revealed that multiple iterations in continual learning facilitate genuine improvement via external feedback, whereas self-feedback alone induces recursive drift. Our data and code are open-source at https://github.com/cxcscmu/SkillLearnBench to enable further studies of automatic skill generation and continual learning techniques.