DocTalk: Scalable Graph-based Dialogue Synthesis for Enhancing LLM Conversational Capabilities
This addresses the problem of enhancing conversational abilities in LLMs for applications like chatbots and virtual assistants, though it is incremental as it builds on existing data synthesis methods.
The paper tackles the mismatch between LLMs' pre-training on continuous prose and their need for multi-turn conversational capabilities by synthesizing conversational data from text corpora, resulting in up to 40% gain in context memory and understanding without compromising base performance.
Large Language Models (LLMs) are increasingly employed in multi-turn conversational tasks, yet their pre-training data predominantly consists of continuous prose, creating a potential mismatch between required capabilities and training paradigms. We introduce a novel approach to address this discrepancy by synthesizing conversational data from existing text corpora. We present a pipeline that transforms a cluster of multiple related documents into an extended multi-turn, multi-topic information-seeking dialogue. Applying our pipeline to Wikipedia articles, we curate DocTalk, a multi-turn pre-training dialogue corpus consisting of over 730k long conversations. We hypothesize that exposure to such synthesized conversational structures during pre-training can enhance the fundamental multi-turn capabilities of LLMs, such as context memory and understanding. Empirically, we show that incorporating DocTalk during pre-training results in up to 40% gain in context memory and understanding, without compromising base performance. DocTalk is available at https://huggingface.co/datasets/AmazonScience/DocTalk.