TERMS-Bench: Diagnosing LLM Negotiation Agents Beyond Deal Rate
For researchers and developers of LLM-based negotiation agents, this work provides a diagnostic framework to identify specific failure modes, moving beyond opaque aggregate rankings.
The paper introduces TERMS-Bench, a Bayesian-game framework for diagnosing LLM negotiation agents beyond aggregate metrics like deal rate. Evaluating 13 LLM agents, it reveals that frontier models saturate deal rate but diverge in surplus extraction, cue use, belief calibration, and compliance, exposing agent-specific bottlenecks masked by prior benchmarks.
Negotiation is a central mechanism of economic exchange, shaping markets, procurement, labor agreements, and resource allocation. It is also a canonical testbed for agentic language models, requiring multi-turn interaction under hidden preferences, strategic communication, and binding constraints. These properties make negotiation hard to evaluate: unlike math or code, it has no intrinsic verifier. Existing LLM negotiation evaluations rely on LLM-vs.-LLM interaction or aggregate outcomes such as deal rate, leaving failures opaque. We introduce Terms-Bench, short for Testbed for Economic Reasoning in Multi-turn Strategy, a Bayesian-game framework that makes the environment itself the verifier by specifying the counterpart's latent type, policy, and payoff structure. We instantiate it in bilateral price negotiation, where the counterpart's private state and simulator policy are hidden from the agent but observable to the evaluator. This turns the counterpart from a black-box opponent into a diagnostic instrument, enabling agent-attributable failure analysis and oracle-reference optimality gaps. Evaluating 13 LLM agents spanning frontier systems from major providers, Terms-Bench turns negotiation evaluation from aggregate ranking into actionable diagnosis: where agents fail, why they fail, and what to strengthen. Empirically, frontier models saturate deal rate yet diverge in surplus extraction, cue use, belief calibration, and compliance, revealing agent-specific bargaining bottlenecks masked by prior benchmarks.