CIRCLE: A Framework for Evaluating AI from a Real-World Lens
This addresses the problem for decision-makers outside the AI stack who lack systematic evidence about AI behavior in real-world contexts, though it is incremental as it builds on existing frameworks like TEVV.
The paper tackles the gap between AI model performance metrics and real-world outcomes by proposing CIRCLE, a six-stage framework that operationalizes validation to translate stakeholder concerns into measurable signals, enabling governance based on materialized effects.
This paper proposes CIRCLE, a six-stage, lifecycle-based framework to bridge the reality gap between model-centric performance metrics and AI's materialized outcomes in deployment. While existing frameworks like MLOps focus on system stability and benchmarks measure abstract capabilities, decision-makers outside the AI stack lack systematic evidence about the behavior of AI technologies under real-world user variability and constraints. CIRCLE operationalizes the Validation phase of TEVV (Test, Evaluation, Verification, and Validation) by formalizing the translation of stakeholder concerns outside the stack into measurable signals. Unlike participatory design, which often remains localized, or algorithmic audits, which are often retrospective, CIRCLE provides a structured, prospective protocol for linking context-sensitive qualitative insights to scalable quantitative metrics. By integrating methods such as field testing, red teaming, and longitudinal studies into a coordinated pipeline, CIRCLE produces systematic knowledge: evidence that is comparable across sites yet sensitive to local context. This can enable governance based on materialized downstream effects rather than theoretical capabilities.