LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions
This addresses the need for open, diverse, and scalable long-context instruction data for AI researchers and developers, offering a novel method to overcome the limitations of proprietary data and costly human annotation.
The paper tackles the problem of generating high-quality long-context instruction data for aligning large language models by introducing LongMagpie, a self-synthesis framework that automatically produces such data without human effort, achieving leading performance on long-context tasks like HELMET, RULER, and Longbench v2 while maintaining competitive results on short-context tasks.
High-quality long-context instruction data is essential for aligning long-context large language models (LLMs). Despite the public release of models like Qwen and Llama, their long-context instruction data remains proprietary. Human annotation is costly and challenging, while template-based synthesis methods limit scale, diversity, and quality. We introduce LongMagpie, a self-synthesis framework that automatically generates large-scale long-context instruction data. Our key insight is that aligned long-context LLMs, when presented with a document followed by special tokens preceding a user turn, auto-regressively generate contextually relevant queries. By harvesting these document-query pairs and the model's responses, LongMagpie produces high-quality instructions without human effort. Experiments on HELMET, RULER, and Longbench v2 demonstrate that LongMagpie achieves leading performance on long-context tasks while maintaining competitive performance on short-context tasks, establishing it as a simple and effective approach for open, diverse, and scalable long-context instruction data synthesis.