Decocted Experience Improves Test-Time Inference in LLM Agents
This work addresses the high cost and suboptimal exploration in LLM agents for complex tasks, offering a complementary approach to test-time scaling, though it is incremental as it builds on existing context-based methods.
The paper tackles the problem of inefficient test-time compute scaling in LLM agents by exploring context construction from experience, showing that decocted experience—extracting and organizing essence from past tasks—improves performance across reasoning and agentic tasks.
There is growing interest in improving LLMs without updating model parameters. One well-established direction is test-time scaling, where increased inference-time computation (e.g., longer reasoning, sampling, or search) is used to improve performance. However, for complex reasoning and agentic tasks, naively scaling test-time compute can substantially increase cost and still lead to wasted budget on suboptimal exploration. In this paper, we explore \emph{context} as a complementary scaling axis for improving LLM performance, and systematically study how to construct better inputs that guide reasoning through \emph{experience}. We show that effective context construction critically depends on \emph{decocted experience}. We present a detailed analysis of experience-augmented agents, studying how to derive context from experience, how performance scales with accumulated experience, what characterizes good context, and which data structures best support context construction. We identify \emph{decocted experience} as a key mechanism for effective context construction: extracting essence from experience, organizing it coherently, and retrieving salient information to build effective context. We validate our findings across reasoning and agentic tasks, including math reasoning, web browsing, and software engineering.