Self-Directed Synthetic Dialogues and Revisions Technical Report
This work addresses a bottleneck for researchers developing open fine-tuning methods by providing a multi-turn synthetic dialogue dataset, though it is incremental as it builds on existing synthetic data and Constitutional AI approaches.
The paper tackles the lack of open multi-turn synthetic dialogue data for fine-tuning language models by introducing SDSD, an experimental dataset of guided conversations generated using open models like DBRX and Llama 2 70B, with revisions based on Constitutional AI principles to create synthetic preference data.
Synthetic data has become an important tool in the fine-tuning of language models to follow instructions and solve complex problems. Nevertheless, the majority of open data to date is often lacking multi-turn data and collected on closed models, limiting progress on advancing open fine-tuning methods. We introduce Self Directed Synthetic Dialogues (SDSD), an experimental dataset consisting of guided conversations of language models talking to themselves. The dataset consists of multi-turn conversations generated with DBRX, Llama 2 70B, and Mistral Large, all instructed to follow a conversation plan generated prior to the conversation. We also explore including principles from Constitutional AI and other related works to create synthetic preference data via revisions to the final conversation turn. We hope this work encourages further exploration in multi-turn data and the use of open models for expanding the impact of synthetic data.