It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers
For practitioners deploying LLM agents, this work challenges the prevailing monotone assumption and provides tier-aware harness selection guidelines, though results are model-specific due to single-model-per-tier evaluation.
The paper tests the assumption that higher-capability LLM agents need less structured harnesses, finding a non-monotone relationship: increased harness verbosity lowers VTSR by 29-38 percentage points for a frontier chat model, while a strict harness achieves the highest VTSR (91.7%) for a frontier reasoning model, contradicting the monotone inverse hypothesis.
A prevalent assumption in LLM agent deployment holds that more structured harnesses universally improve reliability, and that higher-capability models need proportionally less structural guidance -- together implying a monotone inverse relationship between model capability tier and optimal harness complexity. We test this hypothesis through a controlled 432-run experiment crossing six models across four capability tiers with three harness conditions (light, balanced, strict) on HEAT-24, a 24-task synthetic benchmark with git-based workspace verification. Our results refute the monotone inverse relationship on two fronts. First, for the frontier chat model evaluated (Gemini 2.5 Flash), increased harness verbosity lowers VTSR by 29-38 percentage points -- a harness-complexity paradox. Second, for the frontier reasoning model evaluated (Qwen3.5-122B, extended thinking enabled), strict harness achieves the highest VTSR (91.7%) and the lowest latency, the opposite of the prediction. Within the constrained tier, a 2B model (Gemma4:e2B) matches strong-open-tier stability at 91.7% across all harnesses. Because each tier is represented by a single model in this study, these results should be interpreted as model-specific observations; harness sensitivity appears non-monotone across the models evaluated, and depends critically on model type (chat vs. reasoning). We introduce a six-label failure taxonomy showing that format_violation dominates capable-model failures while wrong_file dominates low-capability failures, and we derive practical tier-aware harness selection guidelines.