Evolutionary Context Search for Automated Skill Acquisition
This addresses the challenge of automated skill acquisition for LLMs, offering an efficient alternative to manual prompt engineering or fine-tuning, though it is incremental as it builds on retrieval-augmented generation methods.
The paper tackles the problem of LLMs failing to acquire new knowledge post-deployment by introducing Evolutionary Context Search (ECS), which searches context combinations using accuracy on a development set without weight updates, resulting in improvements of 27% on BackendBench and 7% on τ-bench airline.
Large Language Models cannot reliably acquire new knowledge post-deployment -- even when relevant text resources exist, models fail to transform them into actionable knowledge without retraining. Retrieval-Augmented Generation attempts to bridge this gap by surfacing relevant documents at inference time, yet similarity-based retrieval often fails to identify context that actually improves task performance. We introduce Evolutionary Context Search (ECS), an evolutionary method that searches context combinations using accuracy on a small development set, requiring only inference calls without weight updates. ECS moves beyond semantic similarity to discover non-obvious context pairings that significantly boost performance. Our empirical results show that ECS improves BackendBench by 27\% and $τ$-bench airline by 7\%. The evolved contexts are model-agnostic, as those evolved with Gemini-3-Flash transfer effectively to Claude Sonnet and DeepSeek. This suggests that ECS opens a path toward automated context discovery for skill acquisition -- an efficient alternative to manual prompt engineering or costly fine-tuning.