CLJan 13, 2022
Datasheet for the Pile
arXiv:2201.07311v147 citations
Originality Synthesis-oriented
AI Analysis
This provides a new dataset for training large language models, addressing data scarcity in AI research.
The authors compiled the Pile, an 825 GiB dataset of human-authored text from 22 sources, designed for large-scale language modeling.
This datasheet describes the Pile, a 825 GiB dataset of human-authored text compiled by EleutherAI for use in large-scale language modeling. The Pile is comprised of 22 different text sources, ranging from original scrapes done for this project, to text data made available by the data owners, to third-party scrapes available online.