Acceptance-Test-Driven Evaluation Protocols for Business-Centric LLM Systems
For developers and operators of business-critical LLM systems, this work provides a governance-oriented framework to align probabilistic models with deterministic institutional requirements, addressing the gap between post-hoc benchmarking and operational needs.
The paper proposes an acceptance-test-driven evaluation protocol for LLM systems that translates stakeholder goals into executable behavioral contracts, release gates, and monitoring signals. It introduces a red-train-green lifecycle to ensure safety, reliability, and auditability before system changes are accepted.
Large language model (LLM) applications are increasingly expected to satisfy deterministic institutional requirements while relying on probabilistic generative components. This mismatch makes ordinary post-hoc benchmarking insufficient for systems that must be safe, reliable, auditable, and economically useful. This paper contributes an evaluation-protocol extension for operational LLM systems grounded in acceptance-test-driven development, safety engineering, and business-centric validation. The extension translates stakeholder goals into executable behavioral contracts, release gates, monitoring signals, and evidence artifacts before prompt, model, retrieval, or agent changes are accepted. It adapts the red-green-refactor discipline of test-driven development to a red-train-green lifecycle: first define failing acceptance tests for desired behavior, then improve the LLM system through prompt changes, retrieval design, fine-tuning, guardrails, or data augmentation, and finally release only when multidimensional gates are satisfied. The contribution is a governance-oriented metric stack, reference architecture, and empirical protocol for comparing acceptance-test-driven LLM development against prompt-first and benchmark-after workflows.