COMPASS: Enhancing Agent Long-Horizon Reasoning with Evolving Context
This work addresses the challenge of sustained reasoning in LLM agents for complex tasks, offering a novel framework that enhances performance, though it appears incremental in its approach to context management.
The paper tackles the problem of long-horizon reasoning in LLM agents by addressing context management as a bottleneck, proposing COMPASS, a hierarchical framework that improves accuracy by up to 20% on benchmarks like GAIA, BrowseComp, and Humanity's Last Exam.
Long-horizon tasks that require sustained reasoning and multiple tool interactions remain challenging for LLM agents: small errors compound across steps, and even state-of-the-art models often hallucinate or lose coherence. We identify context management as the central bottleneck -- extended histories cause agents to overlook critical evidence or become distracted by irrelevant information, thus failing to replan or reflect from previous mistakes. To address this, we propose COMPASS (Context-Organized Multi-Agent Planning and Strategy System), a lightweight hierarchical framework that separates tactical execution, strategic oversight, and context organization into three specialized components: (1) a Main Agent that performs reasoning and tool use, (2) a Meta-Thinker that monitors progress and issues strategic interventions, and (3) a Context Manager that maintains concise, relevant progress briefs for different reasoning stages. Across three challenging benchmarks -- GAIA, BrowseComp, and Humanity's Last Exam -- COMPASS improves accuracy by up to 20% relative to both single- and multi-agent baselines. We further introduce a test-time scaling extension that elevates performance to match established DeepResearch agents, and a post-training pipeline that delegates context management to smaller models for enhanced efficiency.