All Required, In Order: Phase-Level Evaluation for AI-Human Dialogue in Healthcare and Beyond
This addresses the gap between technical AI progress and practical healthcare needs by providing a structured evaluation method for clinicians and engineers, though it appears incremental as it builds on existing evaluation frameworks.
The paper tackles the problem of evaluating conversational AI in clinical settings by introducing OIP-SCE, a method that checks if all required clinical obligations are met in the correct order with clear evidence, making complex rules practical and auditable for healthcare applications.
Conversational AI is starting to support real clinical work, but most evaluation methods miss how compliance depends on the full course of a conversation. We introduce Obligatory-Information Phase Structured Compliance Evaluation (OIP-SCE), an evaluation method that checks whether every required clinical obligation is met, in the right order, with clear evidence for clinicians to review. This makes complex rules practical and auditable, helping close the gap between technical progress and what healthcare actually needs. We demonstrate the method in two case studies (respiratory history, benefits verification) and show how phase-level evidence turns policy into shared, actionable steps. By giving clinicians control over what to check and engineers a clear specification to implement, OIP-SCE provides a single, auditable evaluation surface that aligns AI capability with clinical workflow and supports routine, safe use.