D$^3$: Dynamic Directional Graph-Constrained Data Scheduling for LLM Training
This work provides an incremental improvement to data scheduling for large language models by considering sample interactions, which could benefit researchers and practitioners in LLM optimization.
The paper addresses the problem of data scheduling for LLM training by considering directional influences between data samples. It proposes D$^3$, a framework that models these interactions as a dynamic influence graph and optimizes the training order based on this graph, leading to consistent improvements over existing methods in both pre-training and post-training.
Training data plays a central role in large language models (LLMs) optimization, motivating extensive research on data scheduling strategies. Most existing approaches concentrate on adjusting the overall data distribution but neglect the underlying interactions between samples during training. However, we argue that such interactions cannot be overlooked, as real-world data samples frequently exhibit directional influences on each other, making the training order crucial. Intuitively, we can prioritize train-units with greater influence to improves learning efficiency. In this work, we propose $D^3$, a Dynamic Directional graph-constrained Data scheduling framework. $D^3$ formulates the complex interactions among train-units as a dynamic influence graph, where edges represent loss-based dependencies. It then solves a constrained optimization problem over this graph to derive the training order, which ensures that the data sequence respects the evolving information flow throughout training. Our approach is theoretically motivated and yields consistent improvements over existing data scheduling methods across both pre-training and post-training phases. Furthermore, for scalability, $D^3$ also employs an efficient approximation algorithm that keeps the additional computational overhead within a manageable range. For future research, the code is available at https://github.com/xuyj233/D3.