Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Information
This addresses the challenge of automatic evaluation for open-domain dialogues, where multiple valid responses exist, offering a robust metric for researchers and developers in conversational AI, though it is incremental as it builds on existing methods like CVAEs.
The paper tackles the one-to-many problem in open-domain dialogue evaluation by proposing a learning-based metric (CMN) that uses Conditional Variational Autoencoders with Next Sentence Prediction and Mutual Information to assess semantic similarity in latent space, achieving superior performance on two datasets, particularly for responses distant from reference semantics.
The long-standing one-to-many issue of the open-domain dialogues poses significant challenges for automatic evaluation methods, i.e., there may be multiple suitable responses which differ in semantics for a given conversational context. To tackle this challenge, we propose a novel learning-based automatic evaluation metric (CMN), which can robustly evaluate open-domain dialogues by augmenting Conditional Variational Autoencoders (CVAEs) with a Next Sentence Prediction (NSP) objective and employing Mutual Information (MI) to model the semantic similarity of text in the latent space. Experimental results on two open-domain dialogue datasets demonstrate the superiority of our method compared with a wide range of baselines, especially in handling responses which are distant to the golden reference responses in semantics.