ASTRA-bench: Evaluating Tool-Use Agent Reasoning and Action Planning with Personal User Context
This addresses the need for better benchmarks to develop context-aware AI assistants, though it is incremental as it focuses on evaluation rather than a new method.
The paper tackles the problem of evaluating AI agents' ability to handle personal context and multi-step reasoning by introducing ASTRA-bench, a benchmark with 2,413 scenarios that integrates time-evolving personal data and complex intents, revealing significant performance degradation in state-of-the-art models under high-complexity conditions.
Next-generation AI must manage vast personal data, diverse tools, and multi-step reasoning, yet most benchmarks remain context-free and single-turn. We present ASTRA-bench (Assistant Skills in Tool-use, Reasoning \& Action-planning), a benchmark that uniquely unifies time-evolving personal context with an interactive toolbox and complex user intents. Our event-driven pipeline generates 2,413 scenarios across four protagonists, grounded in longitudinal life events and annotated by referential, functional, and informational complexity. Evaluation of state-of-the-art models (e.g., Claude-4.5-Opus, DeepSeek-V3.2) reveals significant performance degradation under high-complexity conditions, with argument generation emerging as the primary bottleneck. These findings expose critical limitations in current agents' ability to ground reasoning within messy personal context and orchestrate reliable multi-step plans. We release ASTRA-bench with a full execution environment and evaluation scripts to provide a diagnostic testbed for developing truly context-aware AI assistants.