Zyda: A 1.3T Dataset for Open Language Modeling
This addresses the data bottleneck for researchers and developers training large language models, though it is incremental as it integrates existing datasets with enhanced processing.
The authors tackled the shortage of open-source datasets for large-scale language model pretraining by introducing Zyda, a 1.3 trillion token dataset under a permissive license, which improves performance of models like those in the Pythia suite compared to other open datasets.
The size of large language models (LLMs) has scaled dramatically in recent years and their computational and data requirements have surged correspondingly. State-of-the-art language models, even at relatively smaller sizes, typically require training on at least a trillion tokens. This rapid advancement has eclipsed the growth of open-source datasets available for large-scale LLM pretraining. In this paper, we introduce Zyda (Zyphra Dataset), a dataset under a permissive license comprising 1.3 trillion tokens, assembled by integrating several major respected open-source datasets into a single, high-quality corpus. We apply rigorous filtering and deduplication processes, both within and across datasets, to maintain and enhance the quality derived from the original datasets. Our evaluations show that Zyda not only competes favorably with other open datasets like Dolma, FineWeb, and RefinedWeb, but also substantially improves the performance of comparable models from the Pythia suite. Our rigorous data processing methods significantly enhance Zyda's effectiveness, outperforming even the best of its constituent datasets when used independently.