CoDi: Conversational Distillation for Grounded Question Answering
This addresses the problem of enabling efficient, on-device conversational AI for specialist applications with incremental improvements in data synthesis.
The paper tackles the challenge of distilling conversational skills into small language models (SLMs) with limited capacity and scarce high-quality data by introducing CoDi, a data distillation framework that synthesizes large-scale, steerable datasets. The result shows that SLMs trained with CoDi-synthesized data achieve performance comparable to human-annotated data and surpass larger models in zero-shot conversational grounded reasoning tasks.
Distilling conversational skills into Small Language Models (SLMs) with approximately 1 billion parameters presents significant challenges. Firstly, SLMs have limited capacity in their model parameters to learn extensive knowledge compared to larger models. Secondly, high-quality conversational datasets are often scarce, small, and domain-specific. Addressing these challenges, we introduce a novel data distillation framework named CoDi (short for Conversational Distillation, pronounced "Cody"), allowing us to synthesize large-scale, assistant-style datasets in a steerable and diverse manner. Specifically, while our framework is task agnostic at its core, we explore and evaluate the potential of CoDi on the task of conversational grounded reasoning for question answering. This is a typical on-device scenario for specialist SLMs, allowing for open-domain model responses, without requiring the model to "memorize" world knowledge in its limited weights. Our evaluations show that SLMs trained with CoDi-synthesized data achieve performance comparable to models trained on human-annotated data in standard metrics. Additionally, when using our framework to generate larger datasets from web data, our models surpass larger, instruction-tuned models in zero-shot conversational grounded reasoning tasks.