Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams
For practitioners deploying LLM agents in dynamic, real-world environments, this work provides a framework and system to sustain self-improvement over time, addressing the brittleness of existing auto-harness methods on fixed benchmarks.
Adaptive Auto-Harness addresses performance degradation in LLM agent systems deployed on open-ended task streams by decomposing the gap to an oracle harness into evolution and adaptation losses, and using a stateful multi-agent evolver, harness tree with solve-time routing, and human-steering hooks. It outperforms five baselines across prediction-market, security-competition, and event-forecasting streams.
Auto-harness systems such as A-Evolve, GEPA, and Meta-Harness improve LLM agents by optimizing prompts, skills, tools, memories, and supporting infrastructure from execution feedback, but they are typically evaluated on fixed offline benchmarks. Real deployments instead present open-ended task streams: histories grow without a fixed endpoint, heterogeneous tasks require different harnesses, and problem distributions shift over time. These challenges make a single repeatedly and densely updated harness brittle, causing performance degradation as accuracy peaks early and then declines. This motivates sustained harness construction with task-wise adaptation. We introduce Adaptive Auto-Harness, a framework and system for such streams. The framework decomposes the gap to an oracle harness into evolution loss and adaptation loss. The system addresses these losses with a stateful multi-agent evolver, a harness tree with solve-time routing, and human-steering hooks for cases where history lacks the needed signal. Across prediction-market, security-competition, and event-forecasting streams, Adaptive Auto-Harness outperforms five existing auto-harness baselines and ablations attribute gains to better construction, routing, or targeted human steering. Code is available in https://github.com/A-EVO-Lab/AdaptiveHarness .