Measuring the `I don't know' Problem through the Lens of Gricean Quantity
This work addresses a specific issue in dialog systems for improving response quality, but it is incremental as it builds on prior work by providing a new diagnostic tool.
The paper tackles the 'I don't know' problem in neural generative dialog models, where systems produce generic responses, by proposing Relative Utterance Quantity (RUQ) based on Grice's maxim of Quantity, finding that baseline models prefer generic responses most of the time but this can be reduced to less than 5% with tuning.
We consider the intrinsic evaluation of neural generative dialog models through the lens of Grice's Maxims of Conversation (1975). Based on the maxim of Quantity (be informative), we propose Relative Utterance Quantity (RUQ) to diagnose the `I don't know' problem, in which a dialog system produces generic responses. The linguistically motivated RUQ diagnostic compares the model score of a generic response to that of the reference response. We find that for reasonable baseline models, `I don't know' is preferred over the reference the majority of the time, but this can be reduced to less than 5% with hyperparameter tuning. RUQ allows for the direct analysis of the `I don't know' problem, which has been addressed but not analyzed by prior work.