LinkQA: Synthesizing Diverse QA from Multiple Seeds Strongly Linked by Knowledge Points
This addresses the data bottleneck for LLM training by synthesizing diverse QA data, though it is an incremental method building on existing graph and diffusion techniques.
The authors tackled the scarcity of high-quality, diverse training data for large language models by proposing LinkSyn, a knowledge point graph-based synthesis framework that generated LinkQA, a 50B-token multi-disciplinary QA dataset. Continual pre-training with LinkQA improved Llama-3 8B by an average of 11.51% on MMLU and CMMLU benchmarks, establishing new state-of-the-art results.
The advancement of large language models (LLMs) struggles with the scarcity of high-quality, diverse training data. To address this limitation, we propose LinkSyn, a novel knowledge point (KP) graph-based synthesis framework that enables flexible control over discipline and difficulty distributions while balancing KP coverage and popularity. LinkSyn extracts KPs from question-answering (QA) seed data and constructs a KP graph to synthesize diverse QA data from multiple seeds strongly linked by KPs and sampled from graph walks. Specifically, LinkSyn incorporates (1) a knowledge distribution value function to guide the adjustment of path sampling probability and balance KP coverage and popularity during graph walks; (2) diffusion-based synthesis via DeepSeek-R1 by leveraging multiple seeds with dense logical associations along each path; and (3) high-difficulty QA enhancement within given disciplines by flexible difficulty adjustments. By executing LinkSyn, we synthesize LinkQA, a diverse multi-disciplinary QA dataset with 50B tokens. Extensive experiments on Llama-3 8B demonstrate that continual pre-training with LinkQA yields an average improvement of $\mathbf{11.51\%}$ on MMLU and CMMLU, establishing new SOTA results. LinkQA consistently enhances performance across model size and initial FLOPs scales.