The 2nd FutureDial Challenge: Dialog Systems with Retrieval Augmented Generation (FutureDial-RAG)
This work addresses the problem of mitigating hallucination in large language models for real-life dialog systems, but it is incremental as it builds upon existing RAG trends by launching a new challenge.
The paper introduces the FutureDial-RAG challenge to promote research on retrieval augmented generation (RAG) for dialog systems, using a dataset of nearly 3000 customer service dialogs and showing that baseline systems find it challenging to perform accurate retrieval and generate coherent responses.
Recently, increasing research interests have focused on retrieval augmented generation (RAG) to mitigate hallucination for large language models (LLMs). Following this trend, we launch the FutureDial-RAG challenge at SLT 2024, which aims at promoting the study of RAG for dialog systems. The challenge builds upon the MobileCS2 dataset, a real-life customer service datasets with nearly 3000 high-quality dialogs containing annotations for knowledge base query and corresponding results. Over the dataset, we define two tasks, track 1 for knowledge retrieval and track 2 for response generation, which are core research questions in dialog systems with RAG. We build baseline systems for the two tracks and design metrics to measure whether the systems can perform accurate retrieval and generate informative and coherent response. The baseline results show that it is very challenging to perform well on the two tasks, which encourages the participating teams and the community to study how to make better use of RAG for real-life dialog systems.