Mid-Training of Large Language Models: A Survey
This provides a foundational resource for researchers and practitioners in AI/ML by systematizing an emerging but previously un-surveyed training paradigm, though it is incremental as a survey rather than a novel method.
The paper tackles the lack of a unified survey on the mid-training stage for large language models, which refines data quality and optimization to improve generalization and capability, and it introduces the first taxonomy and benchmarks to enable structured comparisons.
Large language models (LLMs) are typically developed through large-scale pre-training followed by task-specific fine-tuning. Recent advances highlight the importance of an intermediate mid-training stage, where models undergo multiple annealing-style phases that refine data quality, adapt optimization schedules, and extend context length. This stage mitigates diminishing returns from noisy tokens, stabilizes convergence, and expands model capability in late training. Its effectiveness can be explained through gradient noise scale, the information bottleneck, and curriculum learning, which together promote generalization and abstraction. Despite widespread use in state-of-the-art systems, there has been no prior survey of mid-training as a unified paradigm. We introduce the first taxonomy of LLM mid-training spanning data distribution, learning-rate scheduling, and long-context extension. We distill practical insights, compile evaluation benchmarks, and report gains to enable structured comparisons across models. We also identify open challenges and propose avenues for future research and practice.