Where is the context? -- A critique of recent dialogue datasets
This work highlights dataset limitations for researchers and developers building conversational AI systems, though it is incremental as it critiques rather than introduces new data.
The paper critiques recent dialogue datasets like MultiWOZ 2.1 and Taskmaster-1, identifying issues such as history independence and ambiguous responses, and outlines desiderata for better datasets.
Recent dialogue datasets like MultiWOZ 2.1 and Taskmaster-1 constitute some of the most challenging tasks for present-day dialogue models and, therefore, are widely used for system evaluation. We identify several issues with the above-mentioned datasets, such as history independence, strong knowledge base dependence, and ambiguous system responses. Finally, we outline key desiderata for future datasets that we believe would be more suitable for the construction of conversational artificial intelligence.