Evaluating Task-Oriented Dialogue Consistency through Constraint Satisfaction
This addresses the challenge of ensuring logical coherence and domain accuracy in task-oriented dialogues for AI systems, but it is incremental as it applies an existing CSP method to a new problem.
They tackled the problem of evaluating consistency in task-oriented dialogues by conceptualizing it as a Constraint Satisfaction Problem (CSP), and found that CSP effectively detects inconsistencies, with state-of-the-art LLMs achieving only a 0.15 accuracy rate in consistent re-lexicalization.
Task-oriented dialogues must maintain consistency both within the dialogue itself, ensuring logical coherence across turns, and with the conversational domain, accurately reflecting external knowledge. We propose to conceptualize dialogue consistency as a Constraint Satisfaction Problem (CSP), wherein variables represent segments of the dialogue referencing the conversational domain, and constraints among variables reflect dialogue properties, including linguistic, conversational, and domain-based aspects. To demonstrate the feasibility of the approach, we utilize a CSP solver to detect inconsistencies in dialogues re-lexicalized by an LLM. Our findings indicate that: (i) CSP is effective to detect dialogue inconsistencies; and (ii) consistent dialogue re-lexicalization is challenging for state-of-the-art LLMs, achieving only a 0.15 accuracy rate when compared to a CSP solver. Furthermore, through an ablation study, we reveal that constraints derived from domain knowledge pose the greatest difficulty in being respected. We argue that CSP captures core properties of dialogue consistency that have been poorly considered by approaches based on component pipelines.