Less Data, Faster Training: repeating smaller datasets speeds up learning via sampling biases
For practitioners training neural networks, this work suggests that using smaller datasets with more repetitions can be a proactive strategy to improve training efficiency, especially for reasoning tasks.
The paper shows that training on smaller datasets with more repetitions can be faster and more compute-efficient than using larger datasets, due to sampling biases that promote beneficial layer-wise growth. This effect is observed across multiple tasks, architectures, and optimizers.
This work investigates the ``small-vs-large gap'', where repeating on fewer samples can lead to compute saving during training compared to using a larger dataset. This is observed across algorithmic tasks, architectures and optimizers and cannot be explained using prior theory. We argue that the speedup comes from appropriate layer-wise growth enabled by sampling biases, which is more pronounced when the dataset size is smaller. We provide both theoretical analysis and empirical evidence from various interventions. Our results suggest that using a smaller dataset with more repetitions is not just a fallback strategy under data scarcity, but can be proactively leveraged as a favorable inductive biases for optimization, particularly in reasoning tasks.