DiagFlowBench: Evaluating How Language Models Handle Off-Procedure Inputs in Grounded Diagnostic Dialogue
For developers of grounded dialogue systems, this benchmark exposes a new failure mode where models give plausible but incorrect advice when faced with off-procedure inputs.
DiagFlowBench tests language models on grounded diagnostic dialogue where operator inputs may stray from procedural documentation. Evaluating ten models on 1,676 conversations shows high variability in abstention rates, with models often selecting a plausible but contextually wrong step instead of fabricating facts, revealing a vulnerability in grounding systems.
Language models increasingly serve as advisory systems in maintenance operations. To prevent hallucination, recent systems ground these models in procedural documentation to constrain them to approved steps. In practice, however, operator queries frequently stray from this path, requiring models to recognise out-of-scope inputs mid-conversation, a dynamic that current benchmarks rarely prioritise. We introduce DiagFlowBench, a dataset of 50 industrial diagnostic flowcharts from a consumer manufacturer converted into 1,676 multi-turn conversations that contrast compliant with out-of-scope utterances. Evaluating a panel of ten commercial and open-weight models reveals high variability in abstention rates, with models commonly selecting a real but contextually inadequate step rather than fabricating facts. The inherent plausibility and authority of this mapped but wrong advice exposes a challenging vulnerability for grounding systems.