Towards a Standard, Enterprise-Relevant Agentic AI Benchmark: Lessons from 5.5 billion tokens' worth of agentic AI evaluations
This work addresses the need for reliable, contamination-resistant benchmarks for agentic AI in enterprise settings, though it is incremental as it builds on existing evaluation methods.
The authors tackled the problem of evaluating agentic AI systems for enterprise adoption by developing the Kamiwaza Agentic Merit Index (KAMI) v0.1 benchmark, which processed over 5.5 billion tokens across 35 model configurations and found that traditional benchmark rankings poorly predict practical agentic performance, with newer models not always outperforming older ones on enterprise tasks.
Enterprise adoption of agentic AI systems requires reliable evaluation methods that reflect real-world deployment scenarios. Traditional LLM benchmarks suffer from training data contamination and fail to measure agentic capabilities such as multi-step tool use and decision-making under uncertainty. We present the Kamiwaza Agentic Merit Index (KAMI) v0.1, an enterprise-focused benchmark that addresses both contamination resistance and agentic evaluation. Through 170,000 LLM test items processing over 5.5 billion tokens across 35 model configurations, we demonstrate that traditional benchmark rankings poorly predict practical agentic performance. Notably, newer generation models like Llama 4 or Qwen 3 do not always outperform their older generation variants on enterprise-relevant tasks, contradicting traditional benchmark trends. We also present insights on cost-performance tradeoffs, model-specific behavioral patterns, and the impact of reasoning capabilities on token efficiency -- findings critical for enterprises making deployment decisions.