Beyond Static Snapshots: A Grounded Evaluation Framework for Language Models at the Agentic Frontier
For researchers and practitioners deploying LLMs as agents, this work highlights critical evaluation flaws and offers a practical solution for verifiable-reward domains, though the approach is domain-specific and incremental.
The paper identifies four systematic failures in current LLM evaluation frameworks that make them inadequate for agentic systems, and proposes the Grounded Continuous Evaluation (GCE) framework with ISOPro, a simulation-based system that replaces learned reward models with deterministic verifiers. On a scheduling domain, ISOPro achieves a 3x accuracy improvement over zero-shot baselines using only 0.216% trainable parameters on consumer hardware.
We argue that current evaluation frameworks for large language models (LLMs) suffer from four systematic failures that make them structurally inadequate for assessing deployed, agentic systems: distributional invalidity (evaluation inputs do not reflect real interaction distributions), temporal invalidity (evaluations are post-hoc rather than training-integrated), scope invalidity (evaluations measure single-turn outputs rather than long-horizon trajectories), and process invalidity (evaluations assess outputs rather than reasoning). These failures compound critically in RLHF, where reward models are evaluated under conditions that do not hold during RL training, making reward hacking a predictable consequence of evaluation design rather than a training pathology. We propose the Grounded Continuous Evaluation (GCE) framework and present ISOPro, a simulation-based fine-tuning and evaluation system. ISOPro replaces the learned reward model with a deterministic ground-truth verifier, eliminating reward hacking by construction in verifiable-reward domains, and operates on LoRA adapter weights updatable on CPU, reducing the hardware barrier by an order of magnitude. We validate ISOPro on a resource-constrained scheduling domain with six difficulty tiers, demonstrating capability emergence visible only through continuous evaluation, an implicit curriculum that forms without researcher curation, and a 3x accuracy improvement over zero-shot baselines, all on consumer hardware with 0.216% trainable parameters.