RealClawBench: Live OpenClaw Benchmarks from Real Developer-Agent Sessions
For researchers and developers of AI agents, this benchmark provides a more realistic evaluation of agent capabilities by using real user requests from deployed sessions, addressing the lack of realism in existing benchmarks.
RealClawBench introduces a live benchmark framework built from real OpenClaw developer-agent sessions, converting 281 real user requests into reproducible, automatically scored tasks. Evaluation of 14 models shows the best system solves only 65.8% of tasks, revealing substantial headroom on realistic workloads.
Agent benchmarks should reflect what users actually ask deployed agents to do, yet existing benchmarks often miss key realism properties of real developer-agent sessions. We introduce RealClawBench, a live benchmark framework built from real OpenClaw sessions to capture the distribution, diversity, and real-world difficulty of deployed agent use. Real user requests are challenging to benchmark because they often depend on local execution environments, involve implicit or underspecified intent, and require nontrivial verification. RealClawBench addresses these challenges with two core mechanisms: reconstructed execution environments and deterministic verifiable scorers, which together convert real sessions into reproducible, automatically scored tasks. The resulting release contains 281 executable tasks sampled from a much larger real-session pool while preserving the source distribution, with maximum final-vs-source Jensen-Shannon divergence of 0.0448. Evaluating 14 contemporary models shows that the best system solves only 65.8% of tasks, revealing substantial headroom on realistic developer-agent workloads. By turning real deployed sessions into controlled evaluation instances, RealClawBench provides a practical path toward benchmarks that better measure agent capability in actual use. Code is available at:https://anonymous.4open.science/r/real-claw-bench-582B.